{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Metadata Extraction\n",
    "\n",
    "In this notebook we will demonstrate following:\n",
    "\n",
    "1. RAG using Metadata Extractors.\n",
    "2. Extract Metadata using PydanticProgram."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Installation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install llama-index\n",
    "!pip install llama_index-readers-web"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nest_asyncio\n",
    "\n",
    "nest_asyncio.apply()\n",
    "\n",
    "import os"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup API Key"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "os.environ[\"OPENAI_API_KEY\"] = \"sk-...\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define LLM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.llms.openai import OpenAI\n",
    "from llama_index.core.schema import MetadataMode\n",
    "from llama_index.core import Settings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "llm = OpenAI(temperature=0.1, model=\"gpt-3.5-turbo\", max_tokens=512)\n",
    "Settings.llm = llm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Node Parser and Metadata Extractors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.core.node_parser import TokenTextSplitter\n",
    "from llama_index.core.extractors import (\n",
    "    QuestionsAnsweredExtractor,\n",
    ")\n",
    "\n",
    "node_parser = TokenTextSplitter(\n",
    "    separator=\" \", chunk_size=256, chunk_overlap=128\n",
    ")\n",
    "\n",
    "question_extractor = QuestionsAnsweredExtractor(\n",
    "    questions=3, llm=llm, metadata_mode=MetadataMode.EMBED\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.readers.web import SimpleWebPageReader\n",
    "\n",
    "reader = SimpleWebPageReader(html_to_text=True)\n",
    "docs = reader.load_data(urls=[\"https://eugeneyan.com/writing/llm-patterns/\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# [eugeneyan](/)\n",
      "\n",
      "  * [Start Here](/start-here/ \"Start Here\")\n",
      "  * [Writing](/writing/ \"Writing\")\n",
      "  * [Speaking](/speaking/ \"Speaking\")\n",
      "  * [Prototyping](/prototyping/ \"Prototyping\")\n",
      "  * [About](/about/ \"About\")\n",
      "\n",
      "# Patterns for Building LLM-based Systems & Products\n",
      "\n",
      "[ [llm](/tag/llm/) [engineering](/tag/engineering/)\n",
      "[production](/tag/production/) [🔥](/tag/🔥/) ]  · 66 min read\n",
      "\n",
      "> Discussions on [HackerNews](https://news.ycombinator.com/item?id=36965993),\n",
      "> [Twitter](https://twitter.com/eugeneyan/status/1686531758701899776), and\n",
      "> [LinkedIn](https://www.linkedin.com/posts/eugeneyan_patterns-for-building-\n",
      "> llm-based-systems-activity-7092300473981927424-_wVo)\n",
      "\n",
      "“There is a large class of problems that are easy to imagine and build demos\n",
      "for, but extremely hard to make products out of. For example, self-driving:\n",
      "It’s easy to demo a car self-driving around a block, but making it into a\n",
      "product takes a decade.” -\n",
      "[Karpathy](https://twitter.com/eugeneyan/status/1672692174704766976)\n",
      "\n",
      "This write-up is about practical patterns for integrating large language\n",
      "models (LLMs) into systems & products. We’ll build on academic research,\n",
      "industry resources, and practitioner know-how, and distill them into key ideas\n",
      "and practices.\n",
      "\n",
      "There are seven key patterns. They’re also organized along the spectrum of\n",
      "improving performance vs. reducing cost/risk, and closer to the data vs.\n",
      "closer to the user.\n",
      "\n",
      "  * Evals: To measure performance\n",
      "  * RAG: To add recent, external knowledge\n",
      "  * Fine-tuning: To get better at specific tasks\n",
      "  * Caching: To reduce latency & cost\n",
      "  * Guardrails: To ensure output quality\n",
      "  * Defensive UX: To anticipate & manage errors gracefully\n",
      "  * Collect user feedback: To build our data flywheel\n",
      "\n",
      "(Also see this addendum on [how to match these LLM patterns to potential\n",
      "problems](/writing/llm-problems/).)\n",
      "\n",
      "![Image](/assets/llm-patterns-og.png)\n",
      "\n",
      "LLM patterns: From data to user, from defensive to offensive (see connections\n",
      "between patterns)\n",
      "\n",
      "## Evals: To measure performance\n",
      "\n",
      "Evaluations are a set of measurements used to assess a model’s performance on\n",
      "a task. They include benchmark data and metrics. From a [HackerNews\n",
      "comment](https://news.ycombinator.com/item?id=36789901):\n",
      "\n",
      "> How important evals are to the team is a major differentiator between folks\n",
      "> rushing out hot garbage and those seriously building products in the space.\n",
      "\n",
      "### Why evals?\n",
      "\n",
      "Evals enable us to measure how well our system or product is doing and detect\n",
      "any regressions. (A system or product can be made up of multiple components\n",
      "such as LLMs, prompt templates, retrieved context, and parameters like\n",
      "temperature.) A representative set of evals takes us a step towards measuring\n",
      "system changes at scale. Without evals, we would be flying blind, or would\n",
      "have to visually inspect LLM outputs with each change.\n",
      "\n",
      "### More about evals\n",
      "\n",
      "**There are many benchmarks in the field of language modeling**. Some notable\n",
      "ones are:\n",
      "\n",
      "  * **[MMLU](https://arxiv.org/abs/2009.03300)** : A set of 57 tasks that span elementary math, US history, computer science, law, and more. To perform well, models must possess extensive world knowledge and problem-solving ability.\n",
      "  * **[EleutherAI Eval](https://github.com/EleutherAI/lm-evaluation-harness)** : Unified framework to test models via zero/few-shot settings on 200 tasks. Incorporates a large number of evals including BigBench, MMLU, etc.\n",
      "  * **[HELM](https://arxiv.org/abs/2211.09110)** : Instead of specific tasks and metrics, HELM offers a comprehensive assessment of LLMs by evaluating them across domains. Metrics include accuracy, calibration, robustness, fairness, bias, toxicity, etc. Tasks include Q&A, information retrieval, summarization, text classification, etc.\n",
      "  * **[AlpacaEval](https://github.com/tatsu-lab/alpaca_eval)** : Automated evaluation framework which measures how often a strong LLM (e.g., GPT-4) prefers the output of one model over a reference model. Metrics include win rate, bias, latency, price, variance, etc. Validated to have high agreement with 20k human annotations.\n",
      "\n",
      "We can group metrics into two categories: context-dependent or context-free.\n",
      "\n",
      "  * **Context-dependent** : These take context into account. They’re often proposed for a specific task; repurposing them for other tasks will require some adjustment.\n",
      "  * **Context-free** : These aren’t tied to the context when evaluating generated output; they only compare the output with the provided gold references. As they’re task agnostic, they’re easier to apply to a wide variety of tasks.\n",
      "\n",
      "To get a better sense of these metrics (and their potential shortfalls), we’ll\n",
      "explore a few of the commonly used metrics such as BLEU, ROUGE, BERTScore, and\n",
      "MoverScore.\n",
      "\n",
      "**[BLEU](https://dl.acm.org/doi/10.3115/1073083.1073135) (Bilingual Evaluation\n",
      "Understudy)** is a precision-based metric: It counts the number of n-grams in\n",
      "the generated output that also show up in the reference, and then divides it\n",
      "by the total number of words in the output. It’s predominantly used in machine\n",
      "translation and remains a popular metric due to its cost-effectiveness.\n",
      "\n",
      "First, precision for various values of \\\\(n\\\\) is computed:\n",
      "\n",
      "\\\\[\\text{precision}_n = \\frac{\\sum_{p \\in \\text{output}} \\sum_{\\text{n-gram}\n",
      "\\in p} \\text{Count}_{\\text{clip}} (\\text{n-gram})}{\\sum_{p \\in \\text{output}}\n",
      "\\sum_{\\text{n-gram} \\in p} \\text{Count}(\\text{n-gram})}\\\\]\n",
      "\n",
      "\\\\(Count_{clip}(\\text{n-gram})\\\\) is clipped by the maximum number of times an\n",
      "n-gram appears in any corresponding reference sentence.\n",
      "\n",
      "\\\\[\\text{Count}_{\\text{clip}}(n\\text{-gram}) = \\min \\left(\\text{matched }\n",
      "n\\text{-gram count}, \\max_{r \\in R} \\left(n\\text{-gram count in }\n",
      "r\\right)\\right)\\\\]\n",
      "\n",
      "Once we’ve computed precision at various \\\\(n\\\\), a final BLEU-N score is\n",
      "computed as the geometric mean of all the \\\\(precision_n\\\\) scores.\n",
      "\n",
      "However, since precision relies solely on n-grams and doesn’t consider the\n",
      "length of the generated output, an output containing just one unigram of a\n",
      "common word (like a stop word) would achieve perfect precision. This can be\n",
      "misleading and encourage outputs that contain fewer words to increase BLEU\n",
      "scores. To counter this, a brevity penalty is added to penalize excessively\n",
      "short sentences.\n",
      "\n",
      "\\\\[BP = \\begin{cases} 1 & \\text{if } |p| > |r| \\\\\\ e^{1-\\frac{|r|}{|p|}} &\n",
      "\\text{otherwise} \\end{cases}\\\\]\n",
      "\n",
      "Thus, the final formula is:\n",
      "\n",
      "\\\\[\\text{BLEU-N} = BP \\cdot \\exp\\left(\\sum_{n=1}^{N} W_n\n",
      "\\log(\\text{precision}_n)\\right)\\\\]\n",
      "\n",
      "**[ROUGE](https://aclanthology.org/W04-1013/) (Recall-Oriented Understudy for\n",
      "Gisting Evaluation)**: In contrast to BLEU, ROUGE is recall-oriented. It\n",
      "counts the number of words in the reference that also occur in the output.\n",
      "It’s typically used to assess automatic summarization tasks.\n",
      "\n",
      "There are several ROUGE variants. ROUGE-N is most similar to BLEU in that it\n",
      "also counts the number of matching n-grams between the output and the\n",
      "reference.\n",
      "\n",
      "\\\\[\\text{ROUGE-N} = \\frac{\\sum_{s_r \\in \\text{references}} \\sum_{n\\text{-gram}\n",
      "\\in s_r} \\text{Count}_{\\text{match}} (n\\text{-gram})}{\\sum_{s_r \\in\n",
      "\\text{references}} \\sum_{n\\text{-gram} \\in s_r} \\text{Count}\n",
      "(n\\text{-gram})}\\\\]\n",
      "\n",
      "Other variants include:\n",
      "\n",
      "  * ROUGE-L: This measures the longest common subsequence (LCS) between the output and the reference. It considers sentence-level structure similarity and zeros in on the longest series of co-occurring in-sequence n-grams.\n",
      "  * ROUGE-S: This measures the skip-bigram between the output and reference. Skip-bigrams are pairs of words that maintain their sentence order regardless of the words that might be sandwiched between them.\n",
      "\n",
      "**[BERTScore](https://arxiv.org/abs/1904.09675)** is an embedding-based metric\n",
      "that uses cosine similarity to compare each token or n-gram in the generated\n",
      "output with the reference sentence. There are three components to BERTScore:\n",
      "\n",
      "  * Recall: Average cosine similarity between each token in the reference and its closest match in the generated output.\n",
      "  * Precision: Average cosine similarity between each token in the generated output and its nearest match in the reference.\n",
      "  * F1: Harmonic mean of recall and precision\n",
      "\n",
      "\\\\[Recall_{\\text{BERT}} = \\frac{1}{|r|} \\sum_{i \\in r} \\max_{j \\in p}\n",
      "\\vec{i}^T \\vec{j}, \\quad Precision_{\\text{BERT}} = \\frac{1}{|p|} \\sum_{j \\in\n",
      "p} \\max_{i \\in r} \\vec{i}^T \\vec{j}\\\\] \\\\[\\text{BERTscore} = F_{\\text{BERT}} =\n",
      "\\frac{2 \\cdot P_{\\text{BERT}} \\cdot R_{\\text{BERT}}}{P_{\\text{BERT}} +\n",
      "R_{\\text{BERT}}}\\\\]\n",
      "\n",
      "BERTScore is useful because it can account for synonyms and paraphrasing.\n",
      "Simpler metrics like BLEU and ROUGE can’t do this due to their reliance on\n",
      "exact matches. BERTScore has been shown to have better correlation for tasks\n",
      "such as image captioning and machine translation.\n",
      "\n",
      "**[MoverScore](https://arxiv.org/abs/1909.02622)** also uses contextualized\n",
      "embeddings to compute the distance between tokens in the generated output and\n",
      "reference. But unlike BERTScore, which is based on one-to-one matching (or\n",
      "“hard alignment”) of tokens, MoverScore allows for many-to-one matching (or\n",
      "“soft alignment”).\n",
      "\n",
      "![BERTScore \\(left\\) vs. MoverScore \\(right\\)](/assets/mover-score.jpg)\n",
      "\n",
      "BERTScore (left) vs. MoverScore (right;\n",
      "[source](https://arxiv.org/abs/1909.02622))\n",
      "\n",
      "MoverScore enables the mapping of semantically related words in one sequence\n",
      "to their counterparts in another sequence. It does this by solving a\n",
      "constrained optimization problem that finds the minimum effort to transform\n",
      "one text into another. The idea is to measure the distance that words would\n",
      "have to move to convert one sequence to another.\n",
      "\n",
      "However, there are several pitfalls to using these conventional benchmarks and\n",
      "metrics.\n",
      "\n",
      "First, there’s **poor correlation between these metrics and human judgments.**\n",
      "BLEU, ROUGE, and others have had [negative correlation with how humans\n",
      "evaluate fluency](https://arxiv.org/abs/2008.12009). They also showed moderate\n",
      "to less correlation with human adequacy scores. In particular, BLEU and ROUGE\n",
      "have [low correlation with tasks that require creativity and\n",
      "diversity](https://arxiv.org/abs/2303.16634).\n",
      "\n",
      "Second, these metrics often have **poor adaptability to a wider variety of\n",
      "tasks**. Adopting a metric proposed for one task to another is not always\n",
      "prudent. For example, exact match metrics such as BLEU and ROUGE are a poor\n",
      "fit for tasks like abstractive summarization or dialogue. Since they’re based\n",
      "on n-gram overlap between output and reference, they don’t make sense for a\n",
      "dialogue task where a wide variety of responses are possible. An output can\n",
      "have zero n-gram overlap with the reference but yet be a good response.\n",
      "\n",
      "Third, these metrics have **poor reproducibility**. Even for the same metric,\n",
      "[high variance is reported across different\n",
      "studies](https://arxiv.org/abs/2008.12009), possibly due to variations in\n",
      "human judgment collection or metric parameter settings. Another study of\n",
      "[ROUGE scores](https://aclanthology.org/2023.acl-long.107/) across 2,000\n",
      "studies found that scores were hard to reproduce, difficult to compare, and\n",
      "often incorrect because evals were often conducted with untested, incorrect\n",
      "ROUGE implementations.\n",
      "\n",
      "![Dimensions of model evaluations with ROUGE](/assets/rogue-scores.jpg)\n",
      "\n",
      "Dimensions of model evaluations with ROUGE\n",
      "([source](https://aclanthology.org/2023.acl-long.107/))\n",
      "\n",
      "And even with recent benchmarks such as MMLU, **the same model can get\n",
      "significantly different scores based on the eval implementation**.\n",
      "[Huggingface compared the original MMLU\n",
      "implementation](https://huggingface.co/blog/evaluating-mmlu-leaderboard) with\n",
      "the HELM and EleutherAI implementations and found that the same example could\n",
      "have different prompts across various providers.\n",
      "\n",
      "![Different prompts for the same question across MMLU\n",
      "implementations](/assets/mmlu-prompt.jpg)\n",
      "\n",
      "Different prompts for the same question across MMLU implementations\n",
      "([source](https://huggingface.co/blog/evaluating-mmlu-leaderboard))\n",
      "\n",
      "Furthermore, the evaluation approach differed across all three benchmarks:\n",
      "\n",
      "  * Original MMLU: Compares predicted probabilities on the answers only (A, B, C, D)\n",
      "  * HELM: Uses the next token probabilities from the model and picks the token with the highest probability, even if it’s _not_ one of the options.\n",
      "  * EleutherAI: Computes probability of the full answer sequence (i.e., a letter followed by the answer text) for each answer. Then, pick answer with highest probability.\n",
      "\n",
      "![Different eval for the same question across MMLU\n",
      "implementations](/assets/mmlu-eval.jpg)\n",
      "\n",
      "Different eval for the same question across MMLU implementations\n",
      "([source](https://huggingface.co/blog/evaluating-mmlu-leaderboard))\n",
      "\n",
      "As a result, even for the same eval, both absolute scores and model ranking\n",
      "can fluctuate widely depending on eval implementation. This means that model\n",
      "metrics aren’t truly comparable—even for the same eval—unless the eval’s\n",
      "implementation is identical down to minute details like prompts and\n",
      "tokenization. Similarly, the author of QLoRA found MMLU overly sensitive and\n",
      "concluded: “[do not work with/report or trust MMLU\n",
      "scores](https://twitter.com/Tim_Dettmers/status/1673446047266504704)”.\n",
      "\n",
      "Beyond conventional evals such as those mentioned above, **an emerging trend\n",
      "is to use a strong LLM as a reference-free metric** to evaluate generations\n",
      "from other LLMs. This means we may not need human judgments or gold references\n",
      "for evaluation.\n",
      "\n",
      "**[G-Eval](https://arxiv.org/abs/2303.16634) is a framework that applies\n",
      "LLMs** with Chain-of-Though (CoT) and a form-filling paradigm to **evaluate\n",
      "LLM outputs**. First, they provide a task introduction and evaluation criteria\n",
      "to an LLM and ask it to generate a CoT of evaluation steps. Then, to evaluate\n",
      "coherence in news summarization, they concatenate the prompt, CoT, news\n",
      "article, and summary and ask the LLM to output a score between 1 to 5.\n",
      "Finally, they use the probabilities of the output tokens from the LLM to\n",
      "normalize the score and take their weighted summation as the final result.\n",
      "\n",
      "![Overview of G-Eval](/assets/geval.jpg)\n",
      "\n",
      "Overview of G-Eval ([source](https://arxiv.org/abs/2303.16634))\n",
      "\n",
      "They found that GPT-4 as an evaluator had a high Spearman correlation with\n",
      "human judgments (0.514), outperforming all previous methods. It also\n",
      "outperformed traditional metrics on aspects such as coherence, consistency,\n",
      "fluency, and relevance. On topical chat, it did better than traditional\n",
      "metrics such as ROUGE-L, BLEU-4, and BERTScore across several criteria such as\n",
      "naturalness, coherence, engagingness, and groundedness.\n",
      "\n",
      "**The[Vicuna](https://arxiv.org/abs/2306.05685) paper adopted a similar\n",
      "approach.** They start by defining eight categories (writing, roleplay,\n",
      "extraction, reasoning, math, coding, STEM, and humanities/social science)\n",
      "before developing 10 questions for each category. Next, they generated answers\n",
      "from five chatbots: LLaMA, Alpaca, ChatGPT, Bard, and Vicuna. Finally, they\n",
      "asked GPT-4 to rate the quality of the answers based on helpfulness,\n",
      "relevance, accuracy, and detail.\n",
      "\n",
      "Overall, they found that GPT-4 not only provided consistent scores but could\n",
      "also give detailed explanations for those scores. Under the single answer\n",
      "grading paradigm, GPT-4 had higher agreement with humans (85%) than the humans\n",
      "had amongst themselves (81%). This suggests that GPT-4’s judgment aligns\n",
      "closely with the human evaluators.\n",
      "\n",
      "**[QLoRA](https://arxiv.org/abs/2305.14314) also used an LLM to evaluate\n",
      "another LLM’s output.** They asked GPT-4 to rate the performance of various\n",
      "models against gpt-3.5-turbo on the Vicuna benchmark. Given the responses from\n",
      "gpt-3.5-turbo and another model, GPT-4 was prompted to score both out of 10\n",
      "and explain its ratings. They also measured performance via direct comparisons\n",
      "between models, simplifying the task to a three-class rating scheme that\n",
      "included ties.\n",
      "\n",
      "To validate the automated evaluation, they collected human judgments on the\n",
      "Vicuna benchmark. Using Mechanical Turk, they enlisted two annotators for\n",
      "comparisons to gpt-3.5-turbo, and three annotators for pairwise comparisons.\n",
      "They found that human and GPT-4 ranking of models were largely in agreement,\n",
      "with a Spearman rank correlation of 0.55 at the model level. This provides an\n",
      "additional data point suggesting that LLM-based automated evals could be a\n",
      "cost-effective and reasonable alternative to human evals.\n",
      "\n",
      "### How to apply evals?\n",
      "\n",
      "**Building solid evals should be the starting point** for any LLM-based system\n",
      "or product (as well as conventional machine learning systems).\n",
      "\n",
      "Unfortunately, classical metrics such as BLEU and ROUGE don’t make sense for\n",
      "more complex tasks such as abstractive summarization or dialogue. Furthermore,\n",
      "we’ve seen that benchmarks like MMLU (and metrics like ROUGE) are sensitive to\n",
      "how they’re implemented and measured. And to be candid, unless your LLM system\n",
      "is studying for a school exam, using MMLU as an eval [doesn’t quite make\n",
      "sense](https://twitter.com/Tim_Dettmers/status/1680782418335367169).\n",
      "\n",
      "Thus, instead of using off-the-shelf benchmarks, we can **start by collecting\n",
      "a set of task-specific evals** (i.e., prompt, context, expected outputs as\n",
      "references). These evals will then guide prompt engineering, model selection,\n",
      "fine-tuning, and so on. And as we update our systems, we can run these evals\n",
      "to quickly measure improvements or regressions. Think of it as Eval Driven\n",
      "Development (EDD).\n",
      "\n",
      "In addition to the evaluation dataset, we **also need useful metrics**. They\n",
      "help us distill performance changes into a single number that’s comparable\n",
      "across eval runs. And if we can simplify the problem, we can choose metrics\n",
      "that are easier to compute and interpret.\n",
      "\n",
      "The simplest task is probably classification: If we’re using an LLM for\n",
      "classification-like tasks (e.g., toxicity detection, document categorization)\n",
      "or extractive QA without dialogue, we can rely on standard classification\n",
      "metrics such as recall, precision, PRAUC, etc. If our task has no correct\n",
      "answer but we have references (e.g., machine translation, extractive\n",
      "summarization), we can rely on reference metrics based on matching (BLEU,\n",
      "ROUGE) or semantic similarity (BERTScore, MoverScore).\n",
      "\n",
      "However, these metrics may not work for more open-ended tasks such as\n",
      "abstractive summarization, dialogue, and others. But collecting human\n",
      "judgments can be slow and expensive. Thus, we may opt to lean on **automated\n",
      "evaluations via a strong LLM**.\n",
      "\n",
      "Relative to human judgments which are typically noisy (due to differing biases\n",
      "among annotators), LLM judgments tend to be less noisy (as the bias is more\n",
      "systematic) but more biased. Nonetheless, since we’re aware of these biases,\n",
      "we can mitigate them accordingly:\n",
      "\n",
      "  * Position bias: LLMs tend to favor the response in the first position. To mitigate this, we can evaluate the same pair of responses twice while swapping their order. If the same response is preferred in both orders, we mark it as a win; else, it’s a tie.\n",
      "  * Verbosity bias: LLMs tend to favor longer, wordier responses over more concise ones, even if the latter is clearer and of higher quality. A possible solution is to ensure that comparison responses are similar in length.\n",
      "  * Self-enhancement bias: LLMs have a slight bias towards their own answers. [GPT-4 favors itself with a 10% higher win rate while Claude-v1 favors itself with a 25% higher win rate.](https://arxiv.org/abs/2306.05685) To counter this, don’t use the same LLM for evaluation tasks.\n",
      "\n",
      "Another tip: Rather than asking an LLM for a direct evaluation (via giving a\n",
      "score), try giving it a reference and asking for a comparison. This helps with\n",
      "reducing noise.\n",
      "\n",
      "Finally, sometimes the best eval is human eval aka vibe check. (Not to be\n",
      "confused with the poorly named code evaluation benchmark\n",
      "[HumanEval](https://arxiv.org/abs/2107.03374).) As mentioned in the [Latent\n",
      "Space podcast with MosaicML](https://www.latent.space/p/mosaic-mpt-7b#details)\n",
      "(34th minute):\n",
      "\n",
      "> The vibe-based eval cannot be underrated. … One of our evals was just having\n",
      "> a bunch of prompts and watching the answers as the models trained and see if\n",
      "> they change. Honestly, I don’t really believe that any of these eval metrics\n",
      "> capture what we care about. One of our prompts was “suggest games for a\n",
      "> 3-year-old and a 7-year-old to play” and that was a lot more valuable to see\n",
      "> how the answer changed during the course of training. — Jonathan Frankle\n",
      "\n",
      "Also see this [deep dive into evals](/writing/abstractive/) for abstractive\n",
      "summarization. It covers reference, context, and preference-based metrics, and\n",
      "also discusses hallucination detection.\n",
      "\n",
      "## Retrieval-Augmented Generation: To add knowledge\n",
      "\n",
      "Retrieval-Augmented Generation (RAG) fetches relevant data from outside the\n",
      "foundation model and enhances the input with this data, providing richer\n",
      "context to improve output.\n",
      "\n",
      "### Why RAG?\n",
      "\n",
      "RAG helps reduce hallucination by grounding the model on the retrieved\n",
      "context, thus increasing factuality. In addition, it’s cheaper to keep\n",
      "retrieval indices up-to-date than to continuously pre-train an LLM. This cost\n",
      "efficiency makes it easier to provide LLMs with access to recent data via RAG.\n",
      "Finally, if we need to update or remove data such as biased or toxic\n",
      "documents, it’s more straightforward to update the retrieval index (compared\n",
      "to fine-tuning or prompting an LLM not to generate toxic outputs).\n",
      "\n",
      "In short, RAG applies mature and simpler ideas from the field of information\n",
      "retrieval to support LLM generation. In a [recent Sequoia\n",
      "survey](https://www.sequoiacap.com/article/llm-stack-perspective/), 88% of\n",
      "respondents believe that retrieval will be a key component of their stack.\n",
      "\n",
      "### More about RAG\n",
      "\n",
      "Before diving into RAG, it helps to have a basic understanding of text\n",
      "embeddings. (Feel free to skip this section if you’re familiar with the\n",
      "subject.)\n",
      "\n",
      "A text embedding is a **compressed, abstract representation of text data**\n",
      "where text of arbitrary length can be represented as a fixed-size vector of\n",
      "numbers. It’s usually learned from a corpus of text such as Wikipedia. Think\n",
      "of them as a universal encoding for text, where **similar items are close to\n",
      "each other while dissimilar items are farther apart**.\n",
      "\n",
      "A good embedding is one that does well on a downstream task, such as\n",
      "retrieving similar items. Huggingface’s [Massive Text Embedding Benchmark\n",
      "(MTEB)](https://huggingface.co/spaces/mteb/leaderboard) scores various models\n",
      "on diverse tasks such as classification, clustering, retrieval, summarization,\n",
      "etc.\n",
      "\n",
      "Quick note: While we mainly discuss text embeddings here, embeddings can take\n",
      "many modalities. For example, [CLIP](https://arxiv.org/abs/2103.00020) is\n",
      "multimodal and embeds images and text in the same space, allowing us to find\n",
      "images most similar to an input text. We can also [embed products based on\n",
      "user behavior](/writing/search-query-matching/#supervised-techniques-improves-\n",
      "modeling-of-our-desired-event) (e.g., clicks, purchases) or [graph\n",
      "relationships](/writing/search-query-matching/#self-supervised-techniques-no-\n",
      "need-for-labels).\n",
      "\n",
      "**RAG has its roots in open-domain Q &A.** An early [Meta\n",
      "paper](https://arxiv.org/abs/2005.04611) showed that retrieving relevant\n",
      "documents via TF-IDF and providing them as context to a language model (BERT)\n",
      "improved performance on an open-domain QA task. They converted each task into\n",
      "a cloze statement and queried the language model for the missing token.\n",
      "\n",
      "Following that, **[Dense Passage Retrieval\n",
      "(DPR)](https://arxiv.org/abs/2004.04906)** showed that using dense embeddings\n",
      "(instead of a sparse vector space such as TF-IDF) for document retrieval can\n",
      "outperform strong baselines like Lucene BM25 (65.2% vs. 42.9% for top-5\n",
      "accuracy.) They also showed that higher retrieval precision translates to\n",
      "higher end-to-end QA accuracy, highlighting the importance of upstream\n",
      "retrieval.\n",
      "\n",
      "To learn the DPR embedding, they fine-tuned two independent BERT-based\n",
      "encoders on existing question-answer pairs. The passage encoder (\\\\(E_p\\\\))\n",
      "embeds text passages into vectors while the query encoder (\\\\(E_q\\\\)) embeds\n",
      "questions into vectors. The query embedding is then used to retrieve \\\\(k\\\\)\n",
      "passages that are most similar to the question.\n",
      "\n",
      "They trained the encoders so that the dot-product similarity makes a good\n",
      "ranking function, and optimized the loss function as the negative log-\n",
      "likelihood of the positive passage. The DPR embeddings are optimized for\n",
      "maximum inner product between the question and relevant passage vectors. The\n",
      "goal is to learn a vector space such that pairs of questions and their\n",
      "relevant passages are close together.\n",
      "\n",
      "For inference, they embed all passages (via \\\\(E_p\\\\)) and index them in FAISS\n",
      "offline. Then, given a question at query time, they compute the question\n",
      "embedding (via \\\\(E_q\\\\)), retrieve the top \\\\(k\\\\) passages via approximate\n",
      "nearest neighbors, and provide it to the language model (BERT) that outputs\n",
      "the answer to the question.\n",
      "\n",
      "**[Retrieval Augmented Generation (RAG)](https://arxiv.org/abs/2005.11401)** ,\n",
      "from which this pattern gets its name, highlighted the downsides of pre-\n",
      "trained LLMs. These include not being able to expand or revise memory, not\n",
      "providing insights into generated output, and hallucinations.\n",
      "\n",
      "To address these downsides, they introduced RAG (aka semi-parametric models).\n",
      "Dense vector retrieval serves as the non-parametric component while a pre-\n",
      "trained LLM acts as the parametric component. They reused the DPR encoders to\n",
      "initialize the retriever and build the document index. For the LLM, they used\n",
      "BART, a 400M parameter seq2seq model.\n",
      "\n",
      "![Overview of Retrieval Augmented Generation](/assets/rag.jpg)\n",
      "\n",
      "Overview of Retrieval Augmented Generation\n",
      "([source](https://arxiv.org/abs/2005.11401))\n",
      "\n",
      "During inference, they concatenate the input with the retrieved document.\n",
      "Then, the LLM generates \\\\(\\text{token}_i\\\\) based on the original input, the\n",
      "retrieved document, and the previous \\\\(i-1\\\\) tokens. For generation, they\n",
      "proposed two approaches that vary in how the retrieved passages are used to\n",
      "generate output.\n",
      "\n",
      "In the first approach, RAG-Sequence, the model uses the same document to\n",
      "generate the complete sequence. Thus, for \\\\(k\\\\) retrieved documents, the\n",
      "generator produces an output for each document. Then, the probability of each\n",
      "output sequence is marginalized (sum the probability of each output sequence\n",
      "in \\\\(k\\\\) and weigh it by the probability of each document being retrieved).\n",
      "Finally, the output sequence with the highest probability is selected.\n",
      "\n",
      "On the other hand, RAG-Token can generate each token based on a _different_\n",
      "document. Given \\\\(k\\\\) retrieved documents, the generator produces a\n",
      "distribution for the next output token for each document before marginalizing\n",
      "(aggregating all the individual token distributions.). The process is then\n",
      "repeated for the next token. This means that, for each token generation, it\n",
      "can retrieve a different set of \\\\(k\\\\) relevant documents based on the\n",
      "original input _and_ previously generated tokens. Thus, documents can have\n",
      "different retrieval probabilities and contribute differently to the next\n",
      "generated token.\n",
      "\n",
      "[**Fusion-in-Decoder (FiD)**](https://arxiv.org/abs/2007.01282) also uses\n",
      "retrieval with generative models for open-domain QA. It supports two methods\n",
      "for retrieval, BM25 (Lucene with default parameters) and DPR. FiD is named for\n",
      "how it performs fusion on the retrieved documents in the decoder only.\n",
      "\n",
      "![Overview of Fusion-in-Decoder](/assets/fid.jpg)\n",
      "\n",
      "Overview of Fusion-in-Decoder ([source](https://arxiv.org/abs/2007.01282))\n",
      "\n",
      "For each retrieved passage, the title and passage are concatenated with the\n",
      "question. These pairs are processed independently in the encoder. They also\n",
      "add special tokens such as `question:`, `title:`, and `context:` before their\n",
      "corresponding sections. The decoder attends over the concatenation of these\n",
      "retrieved passages.\n",
      "\n",
      "Because it processes passages independently in the encoder, it can scale to a\n",
      "large number of passages as it only needs to do self-attention over one\n",
      "context at a time. Thus, compute grows linearly (instead of quadratically)\n",
      "with the number of retrieved passages, making it more scalable than\n",
      "alternatives such as RAG-Token. Then, during decoding, the decoder processes\n",
      "the encoded passages jointly, allowing it to better aggregate context across\n",
      "multiple retrieved passages.\n",
      "\n",
      "[**Retrieval-Enhanced Transformer (RETRO)**](https://arxiv.org/abs/2112.04426)\n",
      "adopts a similar pattern where it combines a frozen BERT retriever, a\n",
      "differentiable encoder, and chunked cross-attention to generate output. What’s\n",
      "different is that RETRO does retrieval throughout the entire pre-training\n",
      "stage, and not just during inference. Furthermore, they fetch relevant\n",
      "documents based on chunks of the input. This allows for finer-grained,\n",
      "repeated retrieval during generation instead of only retrieving once per\n",
      "query.\n",
      "\n",
      "For each input chunk (\\\\(C_u\\\\)), the \\\\(k\\\\) retrieved chunks \\\\(RET(C_u)\\\\)\n",
      "are fed into an encoder. The output is the encoded neighbors \\\\(E^{j}_{u}\\\\)\n",
      "where \\\\(E^{j}_{u} = \\text{Encoder}(\\text{RET}(C_{u})^{j}, H_{u}) \\in\n",
      "\\mathbb{R}^{r \\times d_{0}}\\\\). Here, each chunk encoding is conditioned on\n",
      "\\\\(H_u\\\\) (the intermediate activations) and the activations of chunk\n",
      "\\\\(C_u\\\\) through cross-attention layers. In short, the encoding of the\n",
      "retrieved chunks depends on the attended activation of the input chunk.\n",
      "\\\\(E^{j}_{u}\\\\) is then used to condition the generation of the next chunk.\n",
      "\n",
      "![Overview of RETRO](/assets/retro.jpg)\n",
      "\n",
      "Overview of RETRO ([source](https://arxiv.org/abs/2112.04426))\n",
      "\n",
      "During retrieval, RETRO splits the input sequence into chunks of 64 tokens.\n",
      "Then, it finds text similar to the _previous_ chunk to provide context to the\n",
      "_current_ chunk. The retrieval index consists of two contiguous chunks of\n",
      "tokens, \\\\(N\\\\) and \\\\(F\\\\). The former is the neighbor chunk (64 tokens)\n",
      "which is used to compute the key while the latter is the continuation chunk\n",
      "(64 tokens) in the original document.\n",
      "\n",
      "Retrieval is based on approximate \\\\(k\\\\)-nearest neighbors via \\\\(L_2\\\\)\n",
      "distance (euclidean) on BERT embeddings. (Interesting departure from the usual\n",
      "cosine or dot product similarity.) The retrieval index, built on SCaNN, can\n",
      "query a 2T token database in 10ms.\n",
      "\n",
      "They also demonstrated how to RETRO-fit existing baseline models. By freezing\n",
      "the pre-trained weights and only training the chunked cross-attention and\n",
      "neighbor encoder parameters (< 10% of weights for a 7B model), they can\n",
      "enhance transformers with retrieval while only requiring 6M training sequences\n",
      "(3% of pre-training sequences). RETRO-fitted models were able to surpass the\n",
      "performance of baseline models and achieve performance close to that of RETRO\n",
      "trained from scratch.\n",
      "\n",
      "![Performance from RETRO-fitting a pre-trained model](/assets/retrofit.jpg)\n",
      "\n",
      "Performance from RETRO-fitting a pre-trained model\n",
      "([source](https://arxiv.org/abs/2112.04426))\n",
      "\n",
      "**[Internet-augmented LMs](https://arxiv.org/abs/2203.05115)** proposes using\n",
      "a humble “off-the-shelf” search engine to augment LLMs. First, they retrieve a\n",
      "set of relevant documents via Google Search. Since these retrieved documents\n",
      "tend to be long (average length 2,056 words), they chunk them into paragraphs\n",
      "of six sentences each. Finally, they embed the question and paragraphs via TF-\n",
      "IDF and applied cosine similarity to rank the most relevant paragraphs for\n",
      "each query.\n",
      "\n",
      "![Overview of internet-augmented LLMs](/assets/internet-llm.jpg)\n",
      "\n",
      "Overview of internet-augmented LLMs\n",
      "([source](https://arxiv.org/abs/2203.05115))\n",
      "\n",
      "The retrieved paragraphs are used to condition the LLM via few-shot prompting.\n",
      "They adopt the conventional \\\\(k\\\\)-shot prompting (\\\\(k=15\\\\)) from closed-\n",
      "book QA (only providing question-answer pairs) and extend it with an evidence\n",
      "paragraph, such that each context is an evidence, question, and answer\n",
      "triplet.\n",
      "\n",
      "For the generator, they used Gopher, a 280B parameter model trained on 300B\n",
      "tokens. For each question, they generated four candidate answers based on each\n",
      "of the 50 retrieved paragraphs. Finally, they select the best answer by\n",
      "estimating the answer probability via several methods including direct\n",
      "inference, RAG, noisy channel inference, and Product-of-Experts (PoE). PoE\n",
      "consistently performed the best.\n",
      "\n",
      "RAG has also been **applied to non-QA tasks such as code generation**. While\n",
      "**[CodeT5+](https://arxiv.org/abs/2305.07922)** can be used as a standalone\n",
      "generator, when combined with RAG, it significantly outperforms similar models\n",
      "in code generation.\n",
      "\n",
      "To assess the impact of RAG on code generation, they evaluate the model in\n",
      "three settings:\n",
      "\n",
      "  * Retrieval-based: Fetch the top-1 code sample as the prediction\n",
      "  * Generative-only: Output code based on the decoder only\n",
      "  * Retrieval-augmented: Append top-1 code sample to encoder input before code generation via the decoder.\n",
      "\n",
      "![>Overview of RAG for CodeT5+](/assets/codet5.jpg)\n",
      "\n",
      "Overview of RAG for CodeT5+ ([source](https://arxiv.org/abs/2305.07922))\n",
      "\n",
      "As a qualitative example, they showed that retrieved code provides crucial\n",
      "context (e.g., use `urllib3` for an HTTP request) and guides the generative\n",
      "process towards more correct predictions. In contrast, the generative-only\n",
      "approach returns incorrect output that only captures the concepts of\n",
      "“download” and “compress”.\n",
      "\n",
      "**What if we don’t have relevance judgments for query-passage pairs?** Without\n",
      "them, we would not be able to train the bi-encoders that embed the queries and\n",
      "documents in the same embedding space where relevance is represented by the\n",
      "inner product. **[Hypothetical document embeddings\n",
      "(HyDE)](https://arxiv.org/abs/2212.10496)** suggests a solution.\n",
      "\n",
      "![Overview of HyDE](/assets/hyde.jpg)\n",
      "\n",
      "Overview of HyDE ([source](https://arxiv.org/abs/2212.10496))\n",
      "\n",
      "Given a query, HyDE first prompts an LLM, such as InstructGPT, to generate a\n",
      "hypothetical document. Then, an unsupervised encoder, such as Contriver,\n",
      "encodes the document into an embedding vector. Finally, the inner product is\n",
      "computed between the _hypothetical_ document and the corpus, and the most\n",
      "similar _real_ documents are retrieved.\n",
      "\n",
      "The expectation is that the encoder’s dense bottleneck serves as a lossy\n",
      "compressor and the extraneous, non-factual details are excluded via the\n",
      "embedding. This reframes the relevance modeling problem from a representation\n",
      "learning task to a generation task.\n",
      "\n",
      "### How to apply RAG\n",
      "\n",
      "From experience with [Obsidian-Copilot](/writing/obsidian-copilot/), I’ve\n",
      "found that hybrid retrieval (traditional search index + embedding-based\n",
      "search) works better than either alone. There, I complemented classical\n",
      "retrieval (BM25 via OpenSearch) with semantic search (`e5-small-v2`).\n",
      "\n",
      "Why not embedding-based search only? While it’s great in many instances, there\n",
      "are situations where it falls short, such as:\n",
      "\n",
      "  * Searching for a person or object’s name (e.g., Eugene, Kaptir 2.0)\n",
      "  * Searching for an acronym or phrase (e.g., RAG, RLHF)\n",
      "  * Searching for an ID (e.g., `gpt-3.5-turbo`, `titan-xlarge-v1.01`)\n",
      "\n",
      "But keyword search has its limitations too. It only models simple word\n",
      "frequencies and doesn’t capture semantic or correlation information. Thus, it\n",
      "doesn’t deal well with synonyms or hypernyms (i.e., words that represent a\n",
      "generalization). This is where combining it with semantic search is\n",
      "complementary.\n",
      "\n",
      "In addition, with a conventional search index, we can use metadata to refine\n",
      "results. For example, we can use date filters to prioritize newer documents or\n",
      "narrow our search to a specific time period. And if the search is related to\n",
      "e-commerce, filters on average rating or categories are helpful. Finally,\n",
      "having metadata is handy for downstream ranking, such as prioritizing\n",
      "documents that are cited more, or boosting products by their sales volume.\n",
      "\n",
      "**With regard to embeddings** , the seemingly popular approach is to use\n",
      "[`text-embedding-ada-002`](https://openai.com/blog/new-and-improved-embedding-\n",
      "model). Its benefits include ease of use via an API and not having to maintain\n",
      "our own embedding infra or self-host embedding models. Nonetheless, personal\n",
      "experience and anecdotes from others suggest there are better alternatives for\n",
      "retrieval.\n",
      "\n",
      "The OG embedding approaches include Word2vec and\n",
      "[fastText](https://fasttext.cc). FastText is an open-source, lightweight\n",
      "library that enables users to leverage pre-trained embeddings or train new\n",
      "embedding models. It comes with pre-trained embeddings for 157 languages and\n",
      "is extremely fast, even without a GPU. It’s my go-to for early-stage proof of\n",
      "concepts.\n",
      "\n",
      "Another good baseline is [sentence-\n",
      "transformers](https://github.com/UKPLab/sentence-transformers). It makes it\n",
      "simple to compute embeddings for sentences, paragraphs, and even images. It’s\n",
      "based on workhorse transformers such as BERT and RoBERTa and is available in\n",
      "more than 100 languages.\n",
      "\n",
      "More recently, instructor models have shown SOTA performance. During training,\n",
      "these models prepend the task description to the text. Then, when embedding\n",
      "new text, we simply have to describe the task to get task-specific embeddings.\n",
      "(Not that different from instruction tuning for embedding models IMHO.)\n",
      "\n",
      "An example is the [E5](https://arxiv.org/abs/2212.03533) family of models. For\n",
      "open QA and information retrieval, we simply prepend documents in the index\n",
      "with `passage:`, and prepend queries with `query:`. If the task is symmetric\n",
      "(e.g., semantic similarity, paraphrase retrieval) or if we want to use\n",
      "embeddings as features (e.g., classification, clustering), we just use the\n",
      "`query:` prefix.\n",
      "\n",
      "The [Instructor](https://arxiv.org/abs/2212.09741) model takes it a step\n",
      "further, allowing users to customize the prepended prompt: “Represent the\n",
      "`domain` `task_type` for the `task_objective`:” For example, “Represent the\n",
      "Wikipedia document for retrieval:”. (The domain and task objective are\n",
      "optional). This brings the concept of prompt tuning into the field of text\n",
      "embedding.\n",
      "\n",
      "Finally, as of Aug 1st, the top embedding model on the [MTEB\n",
      "Leaderboard](https://huggingface.co/spaces/mteb/leaderboard) is the\n",
      "[GTE](https://huggingface.co/thenlper/gte-large) family of models by Alibaba\n",
      "DAMO Academy. The top performing model’s size is half of the next best model\n",
      "`e5-large-v2` (0.67GB vs 1.34GB). In 2nd position is `gte-base` with a model\n",
      "size of only 0.22GB and embedding dimension of 768. (H/T\n",
      "[Nirant](https://twitter.com/NirantK).)\n",
      "\n",
      "To retrieve documents with low latency at scale, we use approximate nearest\n",
      "neighbors (ANN). It optimizes for retrieval speed and returns the approximate\n",
      "(instead of exact) top \\\\(k\\\\) most similar neighbors, trading off a little\n",
      "accuracy loss for a large speed up.\n",
      "\n",
      "ANN embedding indices are data structures that let us do ANN searches\n",
      "efficiently. At a high level, they build partitions over the embedding space\n",
      "so we can quickly zoom in on the specific space where the query vector is.\n",
      "Some popular techniques include:\n",
      "\n",
      "  * [Locality Sensitive Hashing](https://en.wikipedia.org/wiki/Locality-sensitive_hashing) (LSH): The core idea is to create hash functions so that similar items are likely to end up in the same hash bucket. By only needing to check the relevant buckets, we can perform ANN queries efficiently.\n",
      "  * [Facebook AI Similarity Search](https://github.com/facebookresearch/faiss) (FAISS): It uses a combination of quantization and indexing for efficient retrieval, supports both CPU and GPU, and can handle billions of vectors due to its efficient use of memory.\n",
      "  * [Hierarchical Navigable Small Worlds](https://github.com/nmslib/hnswlib) (HNSW): Inspired by “six degrees of separation”, it builds a hierarchical graph structure that embodies the small world phenomenon. Here, most nodes can be reached from any other node via a minimum number of hops. This structure allows HNSW to initiate queries from broader, coarser approximations and progressively narrow the search at lower levels.\n",
      "  * [Scalable Nearest Neighbors](https://github.com/google-research/google-research/tree/master/scann) (ScaNN): It has a two-step process. First, coarse quantization reduces the search space. Then, fine-grained search is done within the reduced set. Best recall/latency trade-off I’ve seen.\n",
      "\n",
      "When evaluating an ANN index, some factors to consider include:\n",
      "\n",
      "  * Recall: How does it fare against exact nearest neighbors?\n",
      "  * Latency/throughput: How many queries can it handle per second?\n",
      "  * Memory footprint: How much RAM is required to serve an index?\n",
      "  * Ease of adding new items: Can new items be added without having to reindex all documents (LSH) or does the index need to be rebuilt (ScaNN)?\n",
      "\n",
      "No single framework is better than all others in every aspect. Thus, start by\n",
      "defining your functional and non-functional requirements before benchmarking.\n",
      "Personally, I’ve found ScaNN to be outstanding in the recall-latency trade-off\n",
      "(see benchmark graph [here](/writing/real-time-recommendations/#how-to-design-\n",
      "and-implement-an-mvp)).\n",
      "\n",
      "## Fine-tuning: To get better at specific tasks\n",
      "\n",
      "Fine-tuning is the process of taking a pre-trained model (that has already\n",
      "been trained with a vast amount of data) and further refining it on a specific\n",
      "task. The intent is to harness the knowledge that the model has already\n",
      "acquired during its pre-training and apply it to a specific task, usually\n",
      "involving a smaller, task-specific, dataset.\n",
      "\n",
      "The term “fine-tuning” is used loosely and can refer to several concepts such\n",
      "as:\n",
      "\n",
      "  * **Continued pre-training** : With domain-specific data, apply the same pre-training regime (next token prediction, masked language modeling) on the base model.\n",
      "  * **Instruction fine-tuning** : The pre-trained (base) model is fine-tuned on examples of instruction-output pairs to follow instructions, answer questions, be waifu, etc.\n",
      "  * **Single-task fine-tuning** : The pre-trained model is honed for a narrow and specific task such as toxicity detection or summarization, similar to BERT and T5.\n",
      "  * **Reinforcement learning with human feedback (RLHF)** : This combines instruction fine-tuning with reinforcement learning. It requires collecting human preferences (e.g., pairwise comparisons) which are then used to train a reward model. The reward model is then used to further fine-tune the instructed LLM via RL techniques such as proximal policy optimization (PPO).\n",
      "\n",
      "We’ll mainly focus on single-task and instruction fine-tuning here.\n",
      "\n",
      "### Why fine-tuning?\n",
      "\n",
      "Fine-tuning an open LLM is becoming an increasingly viable alternative to\n",
      "using a 3rd-party, cloud-based LLM for several reasons.\n",
      "\n",
      "**Performance & control:** Fine-tuning can improve the performance of an off-\n",
      "the-shelf base model, and may even surpass a 3rd-party LLM. It also provides\n",
      "greater control over LLM behavior, resulting in a more robust system or\n",
      "product. Overall, fine-tuning enables us to build products that are\n",
      "differentiated from simply using 3rd-party or open LLMs.\n",
      "\n",
      "**Modularization:** Single-task fine-tuning lets us to use an army of smaller\n",
      "models that each specialize on their own tasks. Via this setup, a system can\n",
      "be modularized into individual models for tasks like content moderation,\n",
      "extraction, summarization, etc. Also, given that each model only has to focus\n",
      "on a narrow set of tasks, we can get around the alignment tax, where fine-\n",
      "tuning a model on one task reduces performance on other tasks.\n",
      "\n",
      "**Reduced dependencies:** By fine-tuning and hosting our own models, we can\n",
      "reduce legal concerns about proprietary data (e.g., PII, internal documents\n",
      "and code) being exposed to external APIs. It also gets around constraints that\n",
      "come with 3rd-party LLMs such as rate-limiting, high costs, or overly\n",
      "restrictive safety filters. By fine-tuning and hosting our own LLMs, we can\n",
      "ensure data doesn’t leave our network, and can scale throughput as needed.\n",
      "\n",
      "### More about fine-tuning\n",
      "\n",
      "Why do we need to fine-tune a _base_ model? At the risk of oversimplifying,\n",
      "base models are primarily optimized to predict the next word based on the\n",
      "corpus they’re trained on. Hence, they aren’t naturally adept at following\n",
      "instructions or answering questions. When posed a question, they tend to\n",
      "respond with more questions. Thus, we perform instruction fine-tuning so they\n",
      "learn to respond appropriately.\n",
      "\n",
      "However, fine-tuning isn’t without its challenges. First, we **need a\n",
      "significant volume of demonstration data**. For instance, in the [InstructGPT\n",
      "paper](https://arxiv.org/abs/2203.02155), they used 13k instruction-output\n",
      "samples for supervised fine-tuning, 33k output comparisons for reward\n",
      "modeling, and 31k prompts without human labels as input for RLHF.\n",
      "\n",
      "Furthermore, fine-tuning comes with an alignment tax—the process can lead to\n",
      "**lower performance on certain critical tasks**. (There’s no free lunch after\n",
      "all.) The same InstructGPT paper found that RLHF led to performance\n",
      "regressions (relative to the GPT-3 base model) on public NLP tasks like SQuAD,\n",
      "HellaSwag, and WMT 2015 French to English. (A workaround is to have several\n",
      "smaller, specialized models that excel at narrow tasks.)\n",
      "\n",
      "Fine-tuning is similar to the concept of transfer learning. As defined in\n",
      "Wikipedia: “Transfer learning is a technique in machine learning in which\n",
      "knowledge learned from a task is re-used to boost performance on a related\n",
      "task.” Several years ago, transfer learning made it easy for me to apply\n",
      "ResNet models trained on ImageNet to [classify fashion\n",
      "products](/writing/image-categorization-is-now-live/) and [build image\n",
      "search](/writing/image-search-is-now-live/).\n",
      "\n",
      "**[ULMFit](https://arxiv.org/abs/1801.06146)** is one of the earlier papers to\n",
      "apply transfer learning to text. They established the protocol of self-\n",
      "supervised pre-training (on unlabeled data) followed by fine-tuning (on\n",
      "labeled data). They used AWS-LSTM, an LSTM variant with dropout at various\n",
      "gates.\n",
      "\n",
      "![Overview of ULMFit](/assets/ulmfit.jpg)\n",
      "\n",
      "Overview of ULMFit ([source](https://arxiv.org/abs/1801.06146))\n",
      "\n",
      "During pre-training (next word prediction), the model is trained on\n",
      "wikitext-103 which contains 28.6 Wikipedia articles and 103M words. Then,\n",
      "during target task fine-tuning, the LM is fine-tuned with data from the domain\n",
      "of the specific task. Finally, during classifier fine-tuning, the model is\n",
      "augmented with two additional linear blocks and fine-tuned on the target\n",
      "classification tasks which includes sentiment analysis, question\n",
      "classification, and topic classification.\n",
      "\n",
      "Since then, the pre-training followed by fine-tuning paradigm has driven much\n",
      "progress in language modeling. **[Bidirectional Encoder Representations from\n",
      "Transformers (BERT; encoder only)](https://arxiv.org/abs/1810.04805)** was\n",
      "pre-trained on masked language modeling and next sentence prediction on\n",
      "English Wikipedia and BooksCorpus. It was then fine-tuned on task-specific\n",
      "inputs and labels for single-sentence classification, sentence pair\n",
      "classification, single-sentence tagging, and question & answering.\n",
      "\n",
      "![Overview of BERT](/assets/bert.jpg)\n",
      "\n",
      "Overview of BERT ([source](https://arxiv.org/abs/1810.04805))\n",
      "\n",
      "**[Generative Pre-trained Transformers (GPT; decoder only)](https://s3-us-\n",
      "west-2.amazonaws.com/openai-assets/research-covers/language-\n",
      "unsupervised/language_understanding_paper.pdf)** was first pre-trained on\n",
      "BooksCorpus via next token prediction. This was followed by single-task fine-\n",
      "tuning for tasks such as text classification, textual entailment, similarity,\n",
      "and Q&A. Interestingly, they found that including language modeling as an\n",
      "auxiliary objective helped the model generalize and converge faster during\n",
      "training.\n",
      "\n",
      "![Overview of GPT](/assets/gpt.jpg)\n",
      "\n",
      "Overview of GPT ([source](https://s3-us-west-2.amazonaws.com/openai-\n",
      "assets/research-covers/language-unsupervised/language_understanding_paper.pd))\n",
      "\n",
      "**[Text-to-text Transfer Transformer (T5; encoder-\n",
      "decoder)](https://arxiv.org/abs/1910.10683)** was pre-trained on the Colossal\n",
      "Clean Crawled Corpus (C4), a cleaned version of the Common Crawl from April\n",
      "2019. It employed the same denoising objective as BERT, namely masked language\n",
      "modeling. It was then fine-tuned on tasks such as text classification,\n",
      "abstractive summarization, Q&A, and machine translation.\n",
      "\n",
      "![Overview of T5](/assets/t5.jpg)\n",
      "\n",
      "Overview of T5 ([source](https://arxiv.org/abs/1910.10683))\n",
      "\n",
      "But unlike ULMFIt, BERT, and GPT which used different classifier heads for\n",
      "downstream tasks, T5 represented downstream tasks as text-to-text only. For\n",
      "example, a translation task would have input text starting with `Translation\n",
      "English to German:`, while a summarization task might start with `Summarize:`\n",
      "or `TL;DR:`. The prefix essentially became a hyperparameter (first instance of\n",
      "prompt engineering?) This design choice allowed them to use a single fine-\n",
      "tuned model across a variety of downstream tasks.\n",
      "\n",
      "**[InstructGPT](https://arxiv.org/abs/2203.02155)** expanded this idea of\n",
      "single-task fine-tuning to instruction fine-tuning. The base model was GPT-3,\n",
      "pre-trained on internet data including Common Crawl, WebText, Books, and\n",
      "Wikipedia. It then applied supervised fine-tuning on demonstrations of desired\n",
      "behavior (instruction and output). Next, it trained a reward model on the\n",
      "dataset of comparisons. Finally, it optimized the instructed model against the\n",
      "reward model via PPO, with this last stage focusing more on alignment than\n",
      "specific task performance.\n",
      "\n",
      "![Overview of fine-tuning steps in InstructGPT](/assets/instructgpt.jpg)\n",
      "\n",
      "Overview of fine-tuning steps in InstructGPT\n",
      "([source](https://arxiv.org/abs/2203.02155))\n",
      "\n",
      "Next, let’s move from fine-tuned models to fine-tuning techniques.\n",
      "\n",
      "**[Soft prompt tuning](https://arxiv.org/abs/2104.08691)** prepends a\n",
      "trainable tensor to the model’s input embeddings, essentially creating a soft\n",
      "prompt. Unlike discrete text prompts, soft prompts can be learned via\n",
      "backpropagation, meaning they can be fine-tuned to incorporate signals from\n",
      "any number of labeled examples.\n",
      "\n",
      "Next, there’s **[prefix tuning](https://arxiv.org/abs/2101.00190)**. Instead\n",
      "of adding a soft prompt to the model input, it prepends trainable parameters\n",
      "to the hidden states of all transformer blocks. During fine-tuning, the LM’s\n",
      "original parameters are kept frozen while the prefix parameters are updated.\n",
      "\n",
      "![Overview of prefix-tuning](/assets/prefix.jpg)\n",
      "\n",
      "Overview of prefix-tuning ([source](https://arxiv.org/abs/2101.00190))\n",
      "\n",
      "The paper showed that this achieved performance comparable to full fine-tuning\n",
      "despite requiring updates on just 0.1% of parameters. Moreover, in settings\n",
      "with limited data and involved extrapolation to new topics, it outperformed\n",
      "full fine-tuning. One hypothesis is that training fewer parameters helped\n",
      "reduce overfitting on smaller target datasets.\n",
      "\n",
      "There’s also the **[adapter](https://arxiv.org/abs/1902.00751)** technique.\n",
      "This method adds fully connected network layers twice to each transformer\n",
      "block, after the attention layer and after the feed-forward network layer. On\n",
      "GLUE, it’s able to achieve within 0.4% of the performance of full fine-tuning\n",
      "by just adding 3.6% parameters per task.\n",
      "\n",
      "![Overview of adapters](/assets/adapter.jpg)\n",
      "\n",
      "Overview of adapters ([source](https://arxiv.org/abs/1902.00751))\n",
      "\n",
      "**[Low-Rank Adaptation (LoRA)](https://arxiv.org/abs/2106.09685)** is a\n",
      "technique where adapters are designed to be the product of two low-rank\n",
      "matrices. It was inspired by [Aghajanyan et\n",
      "al.](https://arxiv.org/abs/2012.13255) which showed that, when adapting to a\n",
      "specific task, pre-trained language models have a low intrinsic dimension and\n",
      "can still learn efficiently despite a random projection into a smaller\n",
      "subspace. Thus, LoRA hypothesized that weight updates during adaption also\n",
      "have low intrinsic rank.\n",
      "\n",
      "![Overview of LoRA](/assets/lora.jpg)\n",
      "\n",
      "Overview of LoRA ([source](https://arxiv.org/abs/2106.09685))\n",
      "\n",
      "Similar to prefix tuning, they found that LoRA outperformed several baselines\n",
      "including full fine-tuning. Again, the hypothesis is that LoRA, thanks to its\n",
      "reduced rank, provides implicit regularization. In contrast, full fine-tuning,\n",
      "which updates all weights, could be prone to overfitting.\n",
      "\n",
      "**[QLoRA](https://arxiv.org/abs/2305.14314)** builds on the idea of LoRA. But\n",
      "instead of using the full 16-bit model during fine-tuning, it applies a 4-bit\n",
      "quantized model. It introduced several innovations such as 4-bit NormalFloat\n",
      "(to quantize models), double quantization (for additional memory savings), and\n",
      "paged optimizers (that prevent OOM errors by transferring data to CPU RAM when\n",
      "the GPU runs out of memory).\n",
      "\n",
      "![Overview of QLoRA](/assets/qlora.jpg)\n",
      "\n",
      "Overview of QLoRA ([source](https://arxiv.org/abs/2305.14314))\n",
      "\n",
      "As a result, QLoRA reduces the average memory requirements for fine-tuning a\n",
      "65B model from > 780GB memory to a more manageable 48B without degrading\n",
      "runtime or predictive performance compared to a 16-bit fully fine-tuned\n",
      "baseline.\n",
      "\n",
      "(Fun fact: During a meetup with Tim Dettmers, an author of QLoRA, he quipped\n",
      "that double quantization was “a bit of a silly idea but works perfectly.” Hey,\n",
      "if it works, it works.)\n",
      "\n",
      "### How to apply fine-tuning?\n",
      "\n",
      "The first step is to **collect demonstration data/labels**. These could be for\n",
      "straightforward tasks such as document classification, entity extraction, or\n",
      "summarization, or they could be more complex such as Q&A or dialogue. Some\n",
      "ways to collect this data include:\n",
      "\n",
      "  * **Via experts or crowd-sourced human annotators** : While this is expensive and slow, it usually leads to higher-quality data with [good guidelines](/writing/labeling-guidelines/).\n",
      "  * **Via user feedback** : This can be as simple as asking users to select attributes that describe a product, rating LLM responses with thumbs up or down (e.g., ChatGPT), or logging which images users choose to download (e.g., Midjourney).\n",
      "  * **Query larger open models with permissive licenses** : With prompt engineering, we might be able to elicit reasonable demonstration data from a larger model (Falcon 40B Instruct) that can be used to fine-tune a smaller model.\n",
      "  * **Reuse open-source data** : If your task can be framed as a natural language inference (NLI) task, we could fine-tune a model to perform NLI using [MNLI data](https://cims.nyu.edu/~sbowman/multinli/). Then, we can continue fine-tuning the model on internal data to classify inputs as entailment, neutral, or contradiction.\n",
      "\n",
      "Note: Some LLM terms prevent users from using their output to develop other\n",
      "models.\n",
      "\n",
      "  * [OpenAI Terms of Use](https://openai.com/policies/terms-of-use) (Section 2c, iii): You may not use output from the Services to develop models that compete with OpenAI.\n",
      "  * [LLaMA 2 Community License Agreement](https://ai.meta.com/llama/license/) (Section 1b-v): You will not use the Llama Materials or any output or results of the Llama Materials to improve any other large language model (excluding Llama 2 or derivative works thereof).\n",
      "\n",
      "The next step is to **define evaluation metrics**. We’ve discussed this in a\n",
      "previous section.\n",
      "\n",
      "Then, **select a pre-trained model.** There are [several open LLMs with\n",
      "permissive licenses](https://github.com/eugeneyan/open-llms) to choose from.\n",
      "Excluding Llama 2 (since it isn’t fully commercial use), Falcon-40B is known\n",
      "to be the best-performing model. Nonetheless, I’ve found it unwieldy to fine-\n",
      "tune and serve in production given how heavy it is.\n",
      "\n",
      "Instead, I’m inclined to use smaller models like the Falcon-7B. And if we can\n",
      "simplify and frame the task more narrowly, BERT (340M params), RoBERTA (355M\n",
      "params), and BART (406M params) are solid picks for classification and natural\n",
      "language inference tasks. Beyond that, Flan-T5 (770M and 3B variants) is a\n",
      "reliable baseline for translation, abstractive summarization, headline\n",
      "generation, etc.\n",
      "\n",
      "We may also need to **update the model architecture** , such as when the pre-\n",
      "trained model’s architecture doesn’t align with the task. For example, we\n",
      "might need to update the classification heads on BERT or T5 to match our task.\n",
      "Tip: If the task is a simple binary classification task, NLI models can work\n",
      "out of the box. Entailment is mapped to positive, contradiction is mapped to\n",
      "negative, while the neural label can indicate uncertainty.\n",
      "\n",
      "**Then, pick a fine-tuning approach.** LoRA and QLoRA are good places to\n",
      "start. But if your fine-tuning is more intensive, such as continued pre-\n",
      "training on new domain knowledge, you may find full fine-tuning necessary.\n",
      "\n",
      "**Finally, basic hyperparameter tuning.** Generally, most papers focus on\n",
      "learning rate, batch size, and number of epochs (see LoRA, QLoRA). And if\n",
      "we’re using LoRA, we might want to tune the rank parameter (though the QLoRA\n",
      "paper found that different rank and alpha led to similar results). Other\n",
      "hyperparameters include input sequence length, loss type (contrastive loss vs.\n",
      "token match), and data ratios (like the mix of pre-training or demonstration\n",
      "data, or the ratio of positive to negative examples, among others).\n",
      "\n",
      "## Caching: To reduce latency and cost\n",
      "\n",
      "Caching is a technique to store data that has been previously retrieved or\n",
      "computed. This way, future requests for the same data can be served faster. In\n",
      "the space of serving LLM generations, the popularized approach is to cache the\n",
      "LLM response keyed on the embedding of the input request. Then, for each new\n",
      "request, if a semantically similar request is received, we can serve the\n",
      "cached response.\n",
      "\n",
      "For some practitioners, this sounds like “[a disaster waiting to\n",
      "happen.](https://twitter.com/HanchungLee/status/1681146845186363392)” I’m\n",
      "inclined to agree. Thus, I think the key to adopting this pattern is figuring\n",
      "out how to cache safely, instead of solely depending on semantic similarity.\n",
      "\n",
      "### Why caching?\n",
      "\n",
      "Caching can significantly reduce latency for responses that have been served\n",
      "before. In addition, by eliminating the need to compute a response for the\n",
      "same input again and again, we can reduce the number of LLM requests and thus\n",
      "save cost. Also, there are certain use cases that do not support latency on\n",
      "the order of seconds. Thus, pre-computing and caching may be the only way to\n",
      "serve those use cases.\n",
      "\n",
      "### More about caching\n",
      "\n",
      "A cache is a high-speed storage layer that stores a subset of data that’s\n",
      "accessed more frequently. This lets us serve these requests faster via the\n",
      "cache instead of the data’s primary storage (e.g., search index, relational\n",
      "database). Overall, caching enables efficient reuse of previously fetched or\n",
      "computed data. (More about [caching](https://aws.amazon.com/caching/) and\n",
      "[best practices](https://aws.amazon.com/caching/best-practices/).)\n",
      "\n",
      "An example of caching for LLMs is\n",
      "[GPTCache](https://github.com/zilliztech/GPTCache).\n",
      "\n",
      "![Overview of GPTCache](/assets/gptcache.jpg)\n",
      "\n",
      "Overview of GPTCache ([source](https://github.com/zilliztech/GPTCache))\n",
      "\n",
      "When a new request is received:\n",
      "\n",
      "  * Embedding generator: This embeds the request via various models such as OpenAI’s `text-embedding-ada-002`, FastText, Sentence Transformers, and more.\n",
      "  * Similarity evaluator: This computes the similarity of the request via the vector store and then provides a distance metric. The vector store can either be local (FAISS, Hnswlib) or cloud-based. It can also compute similarity via a model.\n",
      "  * Cache storage: If the request is similar, the cached response is fetched and served.\n",
      "  * LLM: If the request isn’t similar enough, it gets passed to the LLM which then generates the result. Finally, the response is served and cached for future use.\n",
      "\n",
      "Redis also shared a [similar\n",
      "example](https://www.youtube.com/live/9VgpXcfJYvw?feature=share&t=1517),\n",
      "mentioning that some teams go as far as precomputing all the queries they\n",
      "anticipate receiving. Then, they set a similarity threshold on which queries\n",
      "are similar enough to warrant a cached response.\n",
      "\n",
      "### How to apply caching?\n",
      "\n",
      "**We should start with having a good understanding of user request patterns**.\n",
      "This allows us to design the cache thoughtfully so it can be applied reliably.\n",
      "\n",
      "First, let’s consider a non-LLM example. Imagine we’re caching product prices\n",
      "for an e-commerce site. During checkout, is it safe to display the (possibly\n",
      "outdated) cached price? Probably not, since the price the customer sees during\n",
      "checkout should be the same as the final amount they’re charged. Caching isn’t\n",
      "appropriate here as we need to ensure consistency for the customer.\n",
      "\n",
      "Now, bringing it back to LLM responses. Imagine we get a request for a summary\n",
      "of “Mission Impossible 2” that’s semantically similar enough to “Mission\n",
      "Impossible 3”. If we’re looking up cache based on semantic similarity, we\n",
      "could serve the wrong response.\n",
      "\n",
      "We also need to **consider if caching is effective for the usage pattern.**\n",
      "One way to quantify this is via the cache hit rate (percentage of requests\n",
      "served directly from the cache). If the usage pattern is uniformly random, the\n",
      "cache would need frequent updates. Thus, the effort to keep the cache up-to-\n",
      "date could negate any benefit a cache has to offer. On the other hand, if the\n",
      "usage follows a power law where a small proportion of unique requests account\n",
      "for the majority of traffic (e.g., search queries, product views), then\n",
      "caching could be an effective strategy.\n",
      "\n",
      "Beyond semantic similarity, we could also explore caching based on:\n",
      "\n",
      "  * **Item IDs:** This applies when we pre-compute [summaries of product reviews](https://www.cnbc.com/2023/06/12/amazon-is-using-generative-ai-to-summarize-product-reviews.html) or generate a summary for an entire movie trilogy.\n",
      "  * **Pairs of Item IDs:** Such as when we generate comparisons between two movies. While this appears to be \\\\(O(N^2)\\\\), in practice, a small number of combinations drive the bulk of traffic, such as comparison between popular movies in a series or genre.\n",
      "  * **Constrained input:** Such as variables like movie genre, director, or lead actor. For example, if a user is looking for movies by a specific director, we could execute a structured query and run it through an LLM to frame the response more eloquently. Another example is [generating code based on drop-down options](https://cheatlayer.com)—if the code has been verified to work, we can cache it for reliable reuse.\n",
      "\n",
      "Also, **caching doesn’t only have to occur on-the-fly.** As Redis shared, we\n",
      "can pre-compute LLM generations offline or asynchronously before serving them.\n",
      "By serving from a cache, we shift the latency from generation (typically\n",
      "seconds) to cache lookup (milliseconds). Pre-computing in batch can also help\n",
      "reduce cost relative to serving in real-time.\n",
      "\n",
      "While the approaches listed here may not be as flexible as semantically\n",
      "caching on natural language inputs, I think it provides a good balance between\n",
      "efficiency and reliability.\n",
      "\n",
      "## Guardrails: To ensure output quality\n",
      "\n",
      "In the context of LLMs, guardrails validate the output of LLMs, ensuring that\n",
      "the output doesn’t just sound good but is also syntactically correct, factual,\n",
      "and free from harmful content. It also includes guarding against adversarial\n",
      "input.\n",
      "\n",
      "### Why guardrails?\n",
      "\n",
      "First, they help ensure that model outputs are reliable and consistent enough\n",
      "to use in production. For example, we may require output to be in a specific\n",
      "JSON schema so that it’s machine-readable, or we need code generated to be\n",
      "executable. Guardrails can help with such syntactic validation.\n",
      "\n",
      "Second, they provide an additional layer of safety and maintain quality\n",
      "control over an LLM’s output. For example, to verify if the content generated\n",
      "is appropriate for serving, we may want to check that the output isn’t\n",
      "harmful, verify it for factual accuracy, or ensure coherence with the context\n",
      "provided.\n",
      "\n",
      "### More about guardrails\n",
      "\n",
      "**One approach is to control the model’s responses via prompts.** For example,\n",
      "Anthropic shared about prompts designed to guide the model toward generating\n",
      "responses that are [helpful, harmless, and\n",
      "honest](https://arxiv.org/abs/2204.05862) (HHH). They found that Python fine-\n",
      "tuning with the HHH prompt led to better performance compared to fine-tuning\n",
      "with RLHF.\n",
      "\n",
      "![Example of HHH prompt](/assets/hhh.jpg)\n",
      "\n",
      "Example of HHH prompt ([source](https://arxiv.org/abs/2204.05862))\n",
      "\n",
      "**A more common approach is to validate the output.** An example is the\n",
      "[Guardrails package](https://github.com/ShreyaR/guardrails). It allows users\n",
      "to add structural, type, and quality requirements on LLM outputs via Pydantic-\n",
      "style validation. And if the check fails, it can trigger corrective action\n",
      "such as filtering on the offending output or regenerating another response.\n",
      "\n",
      "Most of the validation logic is in\n",
      "[`validators.py`](https://github.com/ShreyaR/guardrails/blob/main/guardrails/validators.py).\n",
      "It’s interesting to see how they’re implemented. Broadly speaking, its\n",
      "validators fall into the following categories:\n",
      "\n",
      "  * Single output value validation: This includes ensuring that the output (i) is one of the predefined choices, (ii) has a length within a certain range, (iii) if numeric, falls within an expected range, and (iv) is a complete sentence.\n",
      "  * Syntactic checks: This includes ensuring that generated URLs are valid and reachable, and that Python and SQL code is bug-free.\n",
      "  * Semantic checks: This verifies that the output is aligned with the reference document, or that the extractive summary closely matches the source document. These checks can be done via cosine similarity or fuzzy matching techniques.\n",
      "  * Safety checks: This ensures that the generated output is free of inappropriate language or that the quality of translated text is high.\n",
      "\n",
      "Nvidia’s [NeMo-Guardrails](https://github.com/NVIDIA/NeMo-Guardrails) follows\n",
      "a similar principle but is designed to guide LLM-based conversational systems.\n",
      "Rather than focusing on syntactic guardrails, it emphasizes semantic ones.\n",
      "This includes ensuring that the assistant steers clear of politically charged\n",
      "topics, provides factually correct information, and can detect jailbreaking\n",
      "attempts.\n",
      "\n",
      "Thus, NeMo’s approach is somewhat different: Instead of using more\n",
      "deterministic checks like verifying if a value exists in a list or inspecting\n",
      "code for syntax errors, NeMo leans heavily on using another LLM to validate\n",
      "outputs (inspired by [SelfCheckGPT](https://arxiv.org/abs/2303.08896)).\n",
      "\n",
      "In their example for fact-checking and preventing hallucination, they ask the\n",
      "LLM itself to check whether the most recent output is consistent with the\n",
      "given context. To fact-check, the LLM is queried if the response is true based\n",
      "on the documents retrieved from the knowledge base. To prevent hallucinations,\n",
      "since there isn’t a knowledge base available, they get the LLM to generate\n",
      "multiple alternative completions which serve as the context. The underlying\n",
      "assumption is that if the LLM produces multiple completions that disagree with\n",
      "one another, the original completion is likely a hallucination.\n",
      "\n",
      "The moderation example follows a similar approach: The response is screened\n",
      "for harmful and unethical content via an LLM. Given the nuance of ethics and\n",
      "harmful content, heuristics and conventional machine learning techniques fall\n",
      "short. Thus, an LLM is required for a deeper understanding of the intent and\n",
      "structure of dialogue.\n",
      "\n",
      "Apart from using guardrails to verify the output of LLMs, we can also\n",
      "**directly steer the output to adhere to a specific grammar.** An example of\n",
      "this is Microsoft’s [Guidance](https://github.com/microsoft/guidance). Unlike\n",
      "Guardrails which [imposes JSON schema via a\n",
      "prompt](https://github.com/ShreyaR/guardrails/blob/main/guardrails/constants.xml#L14),\n",
      "Guidance enforces the schema by injecting tokens that make up the structure.\n",
      "\n",
      "We can think of Guidance as a domain-specific language for LLM interactions\n",
      "and output. It draws inspiration from [Handlebars](https://handlebarsjs.com),\n",
      "a popular templating language used in web applications that empowers users to\n",
      "perform variable interpolation and logical control.\n",
      "\n",
      "However, Guidance sets itself apart from regular templating languages by\n",
      "executing linearly. This means it maintains the order of tokens generated.\n",
      "Thus, by inserting tokens that are part of the structure—instead of relying on\n",
      "the LLM to generate them correctly—Guidance can dictate the specific output\n",
      "format. In their examples, they show how to [generate JSON that’s always\n",
      "valid](https://github.com/microsoft/guidance#guaranteeing-valid-syntax-json-\n",
      "example-notebook), [generate complex output\n",
      "formats](https://github.com/microsoft/guidance#rich-output-structure-example-\n",
      "notebook) with multiple keys, ensure that LLMs [play the right\n",
      "roles](https://github.com/microsoft/guidance#role-based-chat-model-example-\n",
      "notebook), and have [agents interact with each\n",
      "other](https://github.com/microsoft/guidance#agents-notebook).\n",
      "\n",
      "They also introduced a concept called [token\n",
      "healing](https://github.com/microsoft/guidance#token-healing-notebook), a\n",
      "useful feature that helps avoid subtle bugs that occur due to tokenization. In\n",
      "simple terms, it rewinds the generation by one token before the end of the\n",
      "prompt and then restricts the first generated token to have a prefix matching\n",
      "the last token in the prompt. This eliminates the need to fret about token\n",
      "boundaries when crafting prompts.\n",
      "\n",
      "### How to apply guardrails?\n",
      "\n",
      "Though the concept of guardrails for LLMs in industry is still nascent, there\n",
      "are a handful of immediately useful and practical strategies we can consider.\n",
      "\n",
      "**Structural guidance:** Apply guidance whenever possible. It provides direct\n",
      "control over outputs and offers a more precise method to ensure that output\n",
      "conforms to a specific structure or format.\n",
      "\n",
      "**Syntactic guardrails:** These include checking if categorical output is\n",
      "within a set of acceptable choices, or if numeric output is within an expected\n",
      "range. Also, if we generate SQL, these can verify its free from syntax errors\n",
      "and also ensure that all columns in the query match the schema. Ditto for\n",
      "generating code (e.g., Python, JavaScript).\n",
      "\n",
      "**Content safety guardrails:** These verify that the output has no harmful or\n",
      "inappropriate content. It can be as simple as checking against the [List of\n",
      "Dirty, Naughty, Obscene, and Otherwise Bad\n",
      "Words](https://github.com/LDNOOBW/List-of-Dirty-Naughty-Obscene-and-Otherwise-\n",
      "Bad-Words) or using [profanity detection](https://pypi.org/project/profanity-\n",
      "check/) models. (It’s [common to run moderation classifiers on\n",
      "output](https://twitter.com/goodside/status/1685023251532320768).) More\n",
      "complex and nuanced output can rely on an LLM evaluator.\n",
      "\n",
      "**Semantic/factuality guardrails:** These confirm that the output is\n",
      "semantically relevant to the input. Say we’re generating a two-sentence\n",
      "summary of a movie based on its synopsis. We can validate if the produced\n",
      "summary is semantically similar to the output, or have (another) LLM ascertain\n",
      "if the summary accurately represents the provided synopsis.\n",
      "\n",
      "**Input guardrails:** These limit the types of input the model will respond\n",
      "to, helping to mitigate the risk of the model responding to inappropriate or\n",
      "adversarial prompts which would lead to generating harmful content. For\n",
      "example, you’ll get an error if you ask Midjourney to generate NSFW content.\n",
      "This can be as straightforward as comparing against a list of strings or using\n",
      "a moderation classifier.\n",
      "\n",
      "![An example of an input guardrail on Midjourney](/assets/input-guardrail.jpg)\n",
      "\n",
      "An example of an input guardrail on Midjourney\n",
      "\n",
      "## Defensive UX: To anticipate & handle errors gracefully\n",
      "\n",
      "Defensive UX is a design strategy that acknowledges that bad things, such as\n",
      "inaccuracies or hallucinations, can happen during user interactions with\n",
      "machine learning or LLM-based products. Thus, the intent is to anticipate and\n",
      "manage these in advance, primarily by guiding user behavior, averting misuse,\n",
      "and handling errors gracefully.\n",
      "\n",
      "### Why defensive UX?\n",
      "\n",
      "Machine learning and LLMs aren’t perfect—they can produce inaccurate output.\n",
      "Also, they respond differently to the same input over time, such as search\n",
      "engines displaying varying results due to personalization, or LLMs generating\n",
      "diverse output on more creative, higher temperature, settings. This can\n",
      "violate the principle of consistency which advocates for a consistent UI and\n",
      "predictable behaviors.\n",
      "\n",
      "Defensive UX can help mitigate the above by providing:\n",
      "\n",
      "  * **Increased accessibility** : By helping users understand how ML/LLM features work and their limitations, defensive UX makes it more accessible and user-friendly.\n",
      "  * **Increased trust** : When users see that the feature can handle difficult scenarios gracefully and doesn’t produce harmful output, they’re likely to trust it more.\n",
      "  * **Better UX** : By designing the system and UX to handle ambiguous situations and errors, defensive UX paves the way for a smoother, more enjoyable user experience.\n",
      "\n",
      "### More about defensive UX\n",
      "\n",
      "To learn more about defensive UX, we can look at Human-AI guidelines from\n",
      "Microsoft, Google, and Apple.\n",
      "\n",
      "**Microsoft’s[Guidelines for Human-AI\n",
      "Interaction](https://www.microsoft.com/en-us/research/publication/guidelines-\n",
      "for-human-ai-interaction/)** is based on a survey of 168 potential guidelines.\n",
      "These were collected from internal and external industry sources, academic\n",
      "literature, and public articles. After combining guidelines that were similar,\n",
      "filtering guidelines that were too vague or too specific or not AI-specific,\n",
      "and a round of heuristic evaluation, they narrowed it down to 18 guidelines.\n",
      "\n",
      "![Guidelines for Human-AI interaction across the user journey](/assets/ms-\n",
      "guidelines.jpg)\n",
      "\n",
      "Guidelines for Human-AI interaction across the user journey\n",
      "([source](https://www.microsoft.com/en-us/research/project/guidelines-for-\n",
      "human-ai-interaction/))\n",
      "\n",
      "These guidelines follow a certain style: Each one is a succinct action rule of\n",
      "3 - 10 words, beginning with a verb. Each rule is accompanied by a one-liner\n",
      "that addresses potential ambiguities. They are organized based on their likely\n",
      "application during user interaction:\n",
      "\n",
      "  * Initially: Make clear what the system can do (G1), make clear how well the system can do what it can do (G2)\n",
      "  * During interaction: Time services based on context (G3), mitigate social biases (G6)\n",
      "  * When wrong: Support efficient dismissal (G8), support efficient correction (G9)\n",
      "  * Over time: Learn from user behavior (G13), provide global controls (G17)\n",
      "\n",
      "**Google’s[People + AI Guidebook](https://pair.withgoogle.com/guidebook/)** is\n",
      "rooted in data and insights drawn from Google’s product team and academic\n",
      "research. In contrast to Microsoft’s guidelines which are organized around the\n",
      "user, Google organizes its guidelines into concepts that a developer needs to\n",
      "keep in mind.\n",
      "\n",
      "There are 23 patterns grouped around common questions that come up during the\n",
      "product development process, including:\n",
      "\n",
      "  * How do I get started with human-centered AI: Determine if the AI adds value, invest early in good data practices (e.g., evals)\n",
      "  * How do I onboard users to new AI features: Make it safe to explore, anchor on familiarity, automate in phases\n",
      "  * How do I help users build trust in my product: Set the right expectations, be transparent, automate more when the risk is low.\n",
      "\n",
      "**Apple’s[Human Interface Guidelines for Machine\n",
      "Learning](https://developer.apple.com/design/human-interface-\n",
      "guidelines/machine-learning)** differs from the bottom-up approach of academic\n",
      "literature and user studies. Instead, its primary source is practitioner\n",
      "knowledge and experience. Thus, it doesn’t include many references or data\n",
      "points, but instead focuses on Apple’s longstanding design principles. This\n",
      "results in a unique perspective that distinguishes it from the other two\n",
      "guidelines.\n",
      "\n",
      "The document focuses on how Apple’s design principles can be applied to ML-\n",
      "infused products, emphasizing aspects of UI rather than model functionality.\n",
      "It starts by asking developers to consider the role of ML in their app and\n",
      "work backwards from the user experience. This includes questions such as\n",
      "whether ML is:\n",
      "\n",
      "  * Critical or complementary: For example, Face ID cannot work without ML but the keyboard can still work without QuickType.\n",
      "  * Proactive or reactive: Siri Suggestions are proactive while autocorrect is reactive.\n",
      "  * Dynamic or static: Recommendations are dynamic while object detection in Photos only improves with each iOS release.\n",
      "\n",
      "It then delves into several patterns, split into inputs and outputs of a\n",
      "system. Inputs focus on explicit feedback, implicit feedback, calibration, and\n",
      "corrections. This section guides the design for how AI products request and\n",
      "process user data and interactions. Outputs focus on mistakes, multiple\n",
      "options, confidence, attribution, and limitations. The intent is to ensure the\n",
      "model’s output is presented in a comprehensible and useful manner.\n",
      "\n",
      "The differences between the three guidelines are insightful. Google has more\n",
      "emphasis on considerations for training data and model development, likely due\n",
      "to its engineering-driven culture. Microsoft has more focus on mental models,\n",
      "likely an artifact of the HCI academic study. Lastly, Apple’s approach centers\n",
      "around providing a seamless UX, a focus likely influenced by its cultural\n",
      "values and principles.\n",
      "\n",
      "### How to apply defensive UX?\n",
      "\n",
      "Here are some patterns based on the guidelines above. (Disclaimer: I’m not a\n",
      "designer.)\n",
      "\n",
      "**Set the right expectations.** This principle is consistent across all three\n",
      "guidelines:\n",
      "\n",
      "  * Microsoft: Make clear how well the system can do what it can do (help the user understand how often the AI system may make mistakes)\n",
      "  * Google: Set the right expectations (be transparent with your users about what your AI-powered product can and cannot do)\n",
      "  * Apple: Help people establish realistic expectations (describe the limitation in marketing material or within the feature’s context)\n",
      "\n",
      "This can be as simple as adding a brief disclaimer above AI-generated results,\n",
      "like those of Bard, or highlighting our app’s limitations on its landing page,\n",
      "like how ChatGPT does it.\n",
      "\n",
      "![Example of a disclaimer on Google Bard results \\(Note: The code provided\n",
      "will not work.\\)](/assets/bard-disclaimer.png)\n",
      "\n",
      "Example of a disclaimer on Google Bard results (Note: `nrows` is not a valid\n",
      "argument.)\n",
      "\n",
      "By being transparent about our product’s capabilities and limitations, we help\n",
      "users calibrate their expectations about its functionality and output. While\n",
      "this may cause users to trust it less in the short run, it helps foster trust\n",
      "in the long run—users are less likely to overestimate our product and\n",
      "subsequently face disappointment.\n",
      "\n",
      "**Enable efficient dismissal.** This is explicitly mentioned as Microsoft’s\n",
      "Guideline 8: Support efficient dismissal (make it easy to dismiss or ignore\n",
      "undesired AI system services).\n",
      "\n",
      "For example, if a user is navigating our site and a chatbot pops up asking if\n",
      "they need help, it should be easy for the user to dismiss the chatbot. This\n",
      "ensures the chatbot doesn’t get in the way, especially on devices with smaller\n",
      "screens. Similarly, GitHub Copilot allows users to conveniently ignore its\n",
      "code suggestions by simply continuing to type. While this may reduce usage of\n",
      "the AI feature in the short term, it prevents it from becoming a nuisance and\n",
      "potentially reducing customer satisfaction in the long term.\n",
      "\n",
      "**Provide attribution.** This is listed in all three guidelines:\n",
      "\n",
      "  * Microsoft: Make clear why the system did what it did (enable the user to access an explanation of why the AI system behaved as it did)\n",
      "  * Google: Add context from human sources (help users appraise your recommendations with input from 3rd-party sources)\n",
      "  * Apple: Consider using attributions to help people distinguish among results\n",
      "\n",
      "Citations are becoming an increasingly common design element. Take BingChat\n",
      "for example. When we make a query, it includes citations, usually from\n",
      "reputable sources, in its responses. This not only shows where the information\n",
      "came from, but also allows users to assess the quality of the sources.\n",
      "Similarly, imagine we’re using an LLM to explain why a user might like a\n",
      "product. Alongside the LLM-generated explanation, we could include a quote\n",
      "from an actual review or mention the product rating.\n",
      "\n",
      "Context from experts and the community also enhances user trust. For example,\n",
      "if a user is seeking recommendations for a hiking trail, mentioning that a\n",
      "suggested trail comes highly recommended by the relevant community can go a\n",
      "long way. It not only adds value to the recommendation but also helps users\n",
      "calibrate trust through the human connection.\n",
      "\n",
      "![Example of attribution via social proof](/assets/social-proof.jpg)\n",
      "\n",
      "Example of attribution via social proof\n",
      "([source](https://pair.withgoogle.com/guidebook/patterns))\n",
      "\n",
      "Finally, Apple’s guidelines include popular attributions such as “Because\n",
      "you’ve read non-fiction”, “New books by authors you’ve read”. These\n",
      "descriptors not only personalize the experience but also provide context,\n",
      "enhancing user understanding and trust.\n",
      "\n",
      "**Anchor on familiarity.** When introducing users to a new AI product or\n",
      "feature, it helps to guide them with familiar UX patterns and features. This\n",
      "makes it easier for users to focus on the main task and start to earn customer\n",
      "trust in our new product. Resist the temptation to showcase new and “magical”\n",
      "features via exotic UI elements.\n",
      "\n",
      "Along a similar vein, chat-based features are becoming more common due to\n",
      "ChatGPT’s growing popularity. For example, chat with your docs, chat to query\n",
      "your data, chat to buy groceries. However, I [question whether chat is the\n",
      "right UX](/writing/llm-ux/) for most user experiences—it just takes too much\n",
      "effort relative to the familiar UX of clicking on text and images.\n",
      "\n",
      "Furthermore, increasing user effort leads to higher expectations that are\n",
      "harder to meet. Netflix shared that users have [higher expectations for\n",
      "recommendations](https://slideslive.com/38934788/a-human-perspective-on-\n",
      "algorithmic-similarity?ref=folder-59726) that result from explicit actions\n",
      "such as search. In general, the more effort a user puts in (e.g., chat,\n",
      "search), the higher the expectations they have. Contrast this with lower-\n",
      "effort interactions such as scrolling over recommendations slates or clicking\n",
      "on a product.\n",
      "\n",
      "Thus, while chat offers more flexibility, it also demands more user effort.\n",
      "Moreover, using a chat box is less intuitive as it lacks signifiers on how\n",
      "users can adjust the output. Overall, I think that sticking with a familiar\n",
      "and constrained UI makes it easier for users to navigate our product; chat\n",
      "should only be considered as a secondary or tertiary option.\n",
      "\n",
      "## Collect user feedback: To build our data flywheel\n",
      "\n",
      "Gathering user feedback allows us to learn their preferences. Specific to LLM\n",
      "products, user feedback contributes to building evals, fine-tuning, and\n",
      "guardrails. If we think about it, data—such as corpus for pre-training,\n",
      "expert-crafted demonstrations, human preferences for reward modeling—is one of\n",
      "the few moats for LLM products. Thus, we want to be deliberately thinking\n",
      "about collecting user feedback when designing our UX.\n",
      "\n",
      "Feedback can be explicit or implicit. Explicit feedback is information users\n",
      "provide in response to a request by our product; implicit feedback is\n",
      "information we learn from user interactions without needing users to\n",
      "deliberately provide feedback.\n",
      "\n",
      "### Why collect user feedback\n",
      "\n",
      "User feedback **helps our models improve**. By learning what users like,\n",
      "dislike, or complain about, we can improve our models to better meet their\n",
      "needs. It also allows us to **adapt to individual preferences**.\n",
      "Recommendation systems are a prime example. As users interact with items, we\n",
      "learn what they like and dislike and better cater to their tastes over time.\n",
      "\n",
      "In addition, the feedback loop helps us **evaluate our system’s overall\n",
      "performance**. While evals can help us measure model/system performance, user\n",
      "feedback offers a concrete measure of user satisfaction and product\n",
      "effectiveness.\n",
      "\n",
      "### How to collect user feedback\n",
      "\n",
      "**Make it easy for users to provide feedback.** This is echoed across all\n",
      "three guidelines:\n",
      "\n",
      "  * Microsoft: Encourage granular feedback (enable the user to provide feedback indicating their preferences during regular interaction with the AI system)\n",
      "  * Google: Let users give feedback (give users the opportunity for real-time teaching, feedback, and error correction)\n",
      "  * Apple: Provide actionable information your app can use to improve the content and experience it presents to people\n",
      "\n",
      "ChatGPT is one such example. Users can indicate thumbs up/down on responses,\n",
      "or choose to regenerate a response if it’s really bad or unhelpful. This is\n",
      "useful feedback on human preferences which can then be used to fine-tune LLMs.\n",
      "\n",
      "Midjourney is another good example. After images are generated, users can\n",
      "generate a new set of images (negative feedback), tweak an image by asking for\n",
      "a variation (positive feedback), or upscale and download the image (strong\n",
      "positive feedback). This enables Midjourney to gather rich comparison data on\n",
      "the outputs generated.\n",
      "\n",
      "![>Example of collecting user feedback as part of the\n",
      "UX](/assets/midjourney.jpg)\n",
      "\n",
      "Example of collecting user feedback as part of the UX\n",
      "\n",
      "**Consider implicit feedback too.** Implicit feedback is information that\n",
      "arises as users interact with our product. Unlike the specific responses we\n",
      "get from explicit feedback, implicit feedback can provide a wide range of data\n",
      "on user behavior and preferences.\n",
      "\n",
      "Copilot-like assistants are a prime example. Users indicate whether a\n",
      "suggestion was helpful by either wholly accepting it (strong positive\n",
      "feedback), accepting and making minor tweaks (positive feedback), or ignoring\n",
      "it (neutral/negative feedback). Alternatively, they may update the comment\n",
      "that led to the generated code, suggesting that the initial code generation\n",
      "didn’t meet their needs.\n",
      "\n",
      "Chatbots, such as ChatGPT and BingChat, are another example. How has daily\n",
      "usage changed over time? If the product is sticky, it suggests that users like\n",
      "it. Also, how long is the average conversation? This can be tricky to\n",
      "interpret: Is a longer conversation better because the conversation was\n",
      "engaging and fruitful? Or is it worse because it took the user longer to get\n",
      "what they needed?\n",
      "\n",
      "## Other patterns common in machine learning\n",
      "\n",
      "Apart from the seven patterns above, there are other patterns in machine\n",
      "learning that are also relevant to LLM systems and products. They include:\n",
      "\n",
      "  * [Data flywheel](/writing/more-patterns/#data-flywheel-to-continuously-improve--build-a-moat): Continuous data collection improves the model and leads to a better user experience. This, in turn, promotes more usage which provides more data to further evaluate and fine-tune models, creating a virtuous cycle.\n",
      "  * [Cascade](/writing/more-patterns/#cascade-to-split-a-problem-into-smaller-problems): Rather than assigning a single, complex task to the LLM, we can simplify and break it down so it only has to handle tasks it excels at, such as reasoning or communicating eloquently. RAG is an example of this. Instead of relying on the LLM to retrieve and rank items based on its internal knowledge, we can augment LLMs with external knowledge and focus on applying the LLM’s reasoning abilities.\n",
      "  * [Monitoring](/writing/practical-guide-to-maintaining-machine-learning/#monitor-models-for-misbehaviour-when-retraining): This helps demonstrate the value added by the AI system, or the lack of it. Someone shared an anecdote of running an LLM-based customer support solution in prod for two weeks before discontinuing it—an A/B test showed that losses were 12x more when using an LLM as a substitute for their support team!\n",
      "\n",
      "(Read more about design patterns for [machine learning code](/writing/design-\n",
      "patterns/) and [systems](/writing/more-patterns/).)\n",
      "\n",
      "Also, here’s what others said:\n",
      "\n",
      "> Separation of concerns/task decomposition- having distinct prompts for\n",
      "> distinct subtasks and chaining them together helps w attention and\n",
      "> reliability (hurts latency). We were having trouble specifying a rigid\n",
      "> output structure AND variable response content so we split up the tasks —\n",
      "> [Erick Enriquez](https://twitter.com/generick_ez/status/1681153738822516736)\n",
      "\n",
      "> A few others that will be needed: role based access control: who can access\n",
      "> what; security: if I’m using a DB with an LLM, how do I ensure that I have\n",
      "> the right security guards —\n",
      "> [Krishna](https://twitter.com/ntkris/status/16812092400299991050)\n",
      "\n",
      "> Consistent output format: setting outputs to a standardized format such as\n",
      "> JSON; Tool augmentation: offload tasks to more specialised, proven, reliable\n",
      "> models — [Paul Tune](https://twitter.com/ptuls/status/1681284873741561857)\n",
      "\n",
      "> Security: mitigate cache poisoning, input validation, mitigate prompt\n",
      "> injection, training data provenance, output with non-vulnerable code,\n",
      "> mitigate malicious input aimed at influencing requests used by tools (AI\n",
      "> Agent), mitigate denial of service (stress test llm), to name a few :) —\n",
      "> [Anderson\n",
      "> Darario](https://www.linkedin.com/feed/update/urn:li:activity:7087089908229558272?commentUrn=urn%3Ali%3Acomment%3A%28activity%3A7087089908229558272%2C7087224131292684288%29)\n",
      "\n",
      "> Another ux/ui related: incentivize users to provide feedback on generated\n",
      "> answers (implicit or explicit). Implicit could be sth like copilot’s ghost\n",
      "> text style, if accepted with TAB, meaning positive feedback etc. — [Wen\n",
      "> Yang](https://www.linkedin.com/feed/update/urn:li:activity:7087089908229558272?commentUrn=urn%3Ali%3Acomment%3A%28activity%3A7087089908229558272%2C7087149792660750336%29)\n",
      "\n",
      "> Great list. I would add consistency checks like self-consistency sampling,\n",
      "> chaining and decomposition of tasks, and the emsembling of multiple model\n",
      "> outputs. Applying each of these almost daily. [Dan\n",
      "> White](https://www.threads.net/@dwhitena/post/Cu3BBaJtoyj/?igshid=OGQ5ZDc2ODk2ZA==)\n",
      "\n",
      "> Guardrails is super relevant for building analytics tools where llm is a\n",
      "> translator from natural to programming language —\n",
      "> [m_voitko](https://www.threads.net/@m_voitko/post/Cu1b4liNwCS/?igshid=OGQ5ZDc2ODk2ZA==)\n",
      "\n",
      "## Conclusion\n",
      "\n",
      "This is the longest post I’ve written by far. If you’re still with me, thank\n",
      "you! I hope you found reading about these patterns helpful, and that the 2x2\n",
      "below makes sense.\n",
      "\n",
      "![LLM patterns across the axis of data to user, and defensive to\n",
      "offensive.](/assets/llm-patterns.png)\n",
      "\n",
      "LLM patterns across the axis of data to user, and defensive to offensive.\n",
      "\n",
      "We’re still so early on the journey towards building LLM-based systems and\n",
      "products. Are there any other key patterns or resources? What have you found\n",
      "useful or not useful? I’d love to hear your experience. **Please[reach\n",
      "out!](https://twitter.com/eugeneyan)**\n",
      "\n",
      "## References\n",
      "\n",
      "Hendrycks, Dan, et al. [“Measuring massive multitask language\n",
      "understanding.”](https://arxiv.org/abs/2009.03300) arXiv preprint\n",
      "arXiv:2009.03300 (2020).\n",
      "\n",
      "Gao, Leo, et al. [“A Framework for Few-Shot Language Model\n",
      "Evaluation.”](https://github.com/EleutherAI/lm-evaluation-harness) v0.0.1,\n",
      "Zenodo, (2021), doi:10.5281/zenodo.5371628.\n",
      "\n",
      "Liang, Percy, et al. [“Holistic evaluation of language\n",
      "models.”](https://arxiv.org/abs/2211.09110) arXiv preprint arXiv:2211.09110\n",
      "(2022).\n",
      "\n",
      "Dubois, Yann, et al. [“AlpacaFarm: A Simulation Framework for Methods That\n",
      "Learn from Human Feedback.”](https://github.com/tatsu-lab/alpaca_eval) (2023)\n",
      "\n",
      "Papineni, Kishore, et al. [“Bleu: a method for automatic evaluation of machine\n",
      "translation.”](https://dl.acm.org/doi/10.3115/1073083.1073135) Proceedings of\n",
      "the 40th annual meeting of the Association for Computational Linguistics.\n",
      "2002.\n",
      "\n",
      "Lin, Chin-Yew. [“Rouge: A package for automatic evaluation of\n",
      "summaries.”](https://aclanthology.org/W04-1013/) Text summarization branches\n",
      "out. 2004.\n",
      "\n",
      "Zhang, Tianyi, et al. [“Bertscore: Evaluating text generation with\n",
      "bert.”](https://arxiv.org/abs/1904.09675) arXiv preprint arXiv:1904.09675\n",
      "(2019).\n",
      "\n",
      "Zhao, Wei, et al. [“MoverScore: Text generation evaluating with contextualized\n",
      "embeddings and earth mover distance.”](https://arxiv.org/abs/1909.02622) arXiv\n",
      "preprint arXiv:1909.02622 (2019).\n",
      "\n",
      "Sai, Ananya B., Akash Kumar Mohankumar, and Mitesh M. Khapra. [“A survey of\n",
      "evaluation metrics used for NLG systems.”](https://arxiv.org/abs/2008.12009)\n",
      "ACM Computing Surveys (CSUR) 55.2 (2022): 1-39.\n",
      "\n",
      "Grusky, Max. [“Rogue Scores.”](https://aclanthology.org/2023.acl-long.107/)\n",
      "Proceedings of the 61st Annual Meeting of the Association for Computational\n",
      "Linguistics (Volume 1: Long Papers). 2023.\n",
      "\n",
      "Liu, Yang, et al. [“Gpteval: Nlg evaluation using gpt-4 with better human\n",
      "alignment.”](https://arxiv.org/abs/2303.16634) arXiv preprint arXiv:2303.16634\n",
      "(2023).\n",
      "\n",
      "Fourrier, Clémentine, et al. [“What’s going on with the Open LLM\n",
      "Leaderboard?”](https://huggingface.co/blog/evaluating-mmlu-leaderboard#whats-\n",
      "going-on-with-the-open-llm-leaderboard) (2023).\n",
      "\n",
      "Zheng, Lianmin, et al. [“Judging LLM-as-a-judge with MT-Bench and Chatbot\n",
      "Arena.”](https://arxiv.org/abs/2306.05685) arXiv preprint arXiv:2306.05685\n",
      "(2023).\n",
      "\n",
      "Dettmers, Tim, et al. [“Qlora: Efficient finetuning of quantized\n",
      "llms.”](https://arxiv.org/abs/2305.14314) arXiv preprint arXiv:2305.14314\n",
      "(2023).\n",
      "\n",
      "Swyx et al. [MPT-7B and The Beginning of\n",
      "Context=Infinity](https://www.latent.space/p/mosaic-mpt-7b#details) (2023).\n",
      "\n",
      "Fradin, Michelle, Reeder, Lauren [“The New Language Model\n",
      "Stack”](https://www.sequoiacap.com/article/llm-stack-perspective/) (2023).\n",
      "\n",
      "Radford, Alec, et al. [“Learning transferable visual models from natural\n",
      "language supervision.”](https://arxiv.org/abs/2103.00020) International\n",
      "conference on machine learning. PMLR, 2021.\n",
      "\n",
      "Yan, Ziyou. [“Search: Query Matching via Lexical, Graph, and Embedding\n",
      "Methods.”](https://eugeneyan.com/writing/search-query-matching/)\n",
      "eugeneyan.com, (2021).\n",
      "\n",
      "Petroni, Fabio, et al. [“How context affects language models’ factual\n",
      "predictions.”](https://arxiv.org/abs/2005.04611) arXiv preprint\n",
      "arXiv:2005.04611 (2020).\n",
      "\n",
      "Karpukhin, Vladimir, et al. [“Dense passage retrieval for open-domain question\n",
      "answering.”](https://arxiv.org/abs/2004.04906) arXiv preprint arXiv:2004.04906\n",
      "(2020).\n",
      "\n",
      "Lewis, Patrick, et al. [“Retrieval-augmented generation for knowledge-\n",
      "intensive nlp tasks.”](https://arxiv.org/abs/2005.11401) Advances in Neural\n",
      "Information Processing Systems 33 (2020): 9459-9474.\n",
      "\n",
      "Izacard, Gautier, and Edouard Grave. [“Leveraging passage retrieval with\n",
      "generative models for open domain question\n",
      "answering.”](https://arxiv.org/abs/2007.01282) arXiv preprint arXiv:2007.01282\n",
      "(2020).\n",
      "\n",
      "Borgeaud, Sebastian, et al. [“Improving language models by retrieving from\n",
      "trillions of tokens.”](https://arxiv.org/abs/2112.04426) International\n",
      "conference on machine learning. PMLR, (2022).\n",
      "\n",
      "Lazaridou, Angeliki, et al. [“Internet-augmented language models through few-\n",
      "shot prompting for open-domain question\n",
      "answering.”](https://arxiv.org/abs/2203.05115) arXiv preprint arXiv:2203.05115\n",
      "(2022).\n",
      "\n",
      "Wang, Yue, et al. [“Codet5+: Open code large language models for code\n",
      "understanding and generation.”](https://arxiv.org/abs/2305.07922) arXiv\n",
      "preprint arXiv:2305.07922 (2023).\n",
      "\n",
      "Gao, Luyu, et al. [“Precise zero-shot dense retrieval without relevance\n",
      "labels.”](https://arxiv.org/abs/2212.10496) arXiv preprint arXiv:2212.10496\n",
      "(2022).\n",
      "\n",
      "Yan, Ziyou. [“Obsidian-Copilot: An Assistant for Writing &\n",
      "Reflecting.”](https://eugeneyan.com/writing/obsidian-copilot/) eugeneyan.com,\n",
      "(2023).\n",
      "\n",
      "Bojanowski, Piotr, et al. [“Enriching word vectors with subword\n",
      "information.”](https://arxiv.org/abs/1607.04606) Transactions of the\n",
      "association for computational linguistics 5 (2017): 135-146.\n",
      "\n",
      "Reimers, Nils, and Iryna Gurevych. [“Making Monolingual Sentence Embeddings\n",
      "Multilingual Using Knowledge Distillation.”](https://arxiv.org/abs/2004.09813)\n",
      "Proceedings of the 2020 Conference on Empirical Methods in Natural Language\n",
      "Processing, Association for Computational Linguistics, (2020).\n",
      "\n",
      "Wang, Liang, et al. [“Text embeddings by weakly-supervised contrastive pre-\n",
      "training.”](https://arxiv.org/abs/2212.03533) arXiv preprint arXiv:2212.03533\n",
      "(2022).\n",
      "\n",
      "Su, Hongjin, et al. [“One embedder, any task: Instruction-finetuned text\n",
      "embeddings.”](https://arxiv.org/abs/2212.09741) arXiv preprint\n",
      "arXiv:2212.09741 (2022).\n",
      "\n",
      "Johnson, Jeff, et al. [“Billion-Scale Similarity Search with\n",
      "GPUs.”](https://arxiv.org/abs/1702.08734) IEEE Transactions on Big Data, vol.\n",
      "7, no. 3, IEEE, 2019, pp. 535–47.\n",
      "\n",
      "Malkov, Yu A., and Dmitry A. Yashunin. [“Efficient and Robust Approximate\n",
      "Nearest Neighbor Search Using Hierarchical Navigable Small World\n",
      "Graphs.”](https://arxiv.org/abs/1603.09320) IEEE Transactions on Pattern\n",
      "Analysis and Machine Intelligence, vol. 42, no. 4, IEEE, 2018, pp. 824–36.\n",
      "\n",
      "Guo, Ruiqi, et al. [“Accelerating Large-Scale Inference with Anisotropic\n",
      "Vector Quantization.”](https://arxiv.org/abs/1908.10396.) International\n",
      "Conference on Machine Learning, (2020)\n",
      "\n",
      "Ouyang, Long, et al. [“Training language models to follow instructions with\n",
      "human feedback.”](https://arxiv.org/abs/2203.02155) Advances in Neural\n",
      "Information Processing Systems 35 (2022): 27730-27744.\n",
      "\n",
      "Howard, Jeremy, and Sebastian Ruder. [“Universal language model fine-tuning\n",
      "for text classification.”](https://arxiv.org/abs/1801.06146) arXiv preprint\n",
      "arXiv:1801.06146 (2018).\n",
      "\n",
      "Devlin, Jacob, et al. [“Bert: Pre-training of deep bidirectional transformers\n",
      "for language understanding.”](https://arxiv.org/abs/1810.04805) arXiv preprint\n",
      "arXiv:1810.04805 (2018).\n",
      "\n",
      "Radford, Alec, et al. [“Improving language understanding with unsupervised\n",
      "learning.”](https://openai.com/research/language-unsupervised) (2018).\n",
      "\n",
      "Raffel, Colin, et al. [“Exploring the limits of transfer learning with a\n",
      "unified text-to-text transformer.”](https://arxiv.org/abs/1910.10683) The\n",
      "Journal of Machine Learning Research 21.1 (2020): 5485-5551.\n",
      "\n",
      "Lester, Brian, Rami Al-Rfou, and Noah Constant. [“The power of scale for\n",
      "parameter-efficient prompt tuning.”](https://arxiv.org/abs/2104.08691) arXiv\n",
      "preprint arXiv:2104.08691 (2021).\n",
      "\n",
      "Li, Xiang Lisa, and Percy Liang. [“Prefix-tuning: Optimizing continuous\n",
      "prompts for generation.”](https://arxiv.org/abs/2101.00190) arXiv preprint\n",
      "arXiv:2101.00190 (2021).\n",
      "\n",
      "Houlsby, Neil, et al. [“Parameter-efficient transfer learning for\n",
      "NLP.”](https://arxiv.org/abs/1902.00751) International Conference on Machine\n",
      "Learning. PMLR, 2019.\n",
      "\n",
      "Hu, Edward J., et al. [“Lora: Low-rank adaptation of large language\n",
      "models.”](https://arxiv.org/abs/2106.09685) arXiv preprint arXiv:2106.09685\n",
      "(2021).\n",
      "\n",
      "Dettmers, Tim, et al. [“Qlora: Efficient finetuning of quantized\n",
      "llms.”](https://arxiv.org/abs/2305.14314) arXiv preprint arXiv:2305.14314\n",
      "(2023).\n",
      "\n",
      "Williams, Adina, et al. [“A Broad-Coverage Challenge Corpus for Sentence\n",
      "Understanding through Inference.”](https://cims.nyu.edu/~sbowman/multinli/)\n",
      "Proceedings of the 2018 Conference of the North American Chapter of the\n",
      "Association for Computational Linguistics: Human Language Technologies, Volume\n",
      "1 (Long Papers), Association for Computational Linguistics, (2018).\n",
      "\n",
      "[GPTCache](https://github.com/zilliztech/GPTCache) (2023).\n",
      "\n",
      "Bai, Yuntao, et al. [“Training a helpful and harmless assistant with\n",
      "reinforcement learning from human\n",
      "feedback.”](https://arxiv.org/abs/2204.05862) arXiv preprint arXiv:2204.05862\n",
      "(2022).\n",
      "\n",
      "[Guardrails](https://github.com/ShreyaR/guardrails) (2023)\n",
      "\n",
      "[NeMo-Guardrails](https://github.com/NVIDIA/NeMo-Guardrails) (2023)\n",
      "\n",
      "Manakul, Potsawee, Adian Liusie, and Mark JF Gales. [“Selfcheckgpt: Zero-\n",
      "resource black-box hallucination detection for generative large language\n",
      "models.”](https://arxiv.org/abs/2303.08896) arXiv preprint arXiv:2303.08896\n",
      "(2023).\n",
      "\n",
      "[Guidance](https://github.com/microsoft/guidance) (2023).\n",
      "\n",
      "Amershi, Saleema, et al. [“Guidelines for human-AI\n",
      "interaction.”](https://www.microsoft.com/en-\n",
      "us/research/publication/guidelines-for-human-ai-interaction/) Proceedings of\n",
      "the 2019 chi conference on human factors in computing systems. 2019.\n",
      "\n",
      "[People + AI Guidebook](https://pair.withgoogle.com/guidebook/) (2023).\n",
      "\n",
      "[Human Interface Guidelines for Machine\n",
      "Learning](https://developer.apple.com/design/human-interface-\n",
      "guidelines/machine-learning) (2023).\n",
      "\n",
      "Schendel, Zachary A., Faraz Farzin, and Siddhi Sundar. [“A Human Perspective\n",
      "on Algorithmic Similarity.”](https://slideslive.com/38934788/a-human-\n",
      "perspective-on-algorithmic-similarity?ref=folder-59726) Proceedings of the\n",
      "14th ACM Conference on Recommender Systems. 2020.\n",
      "\n",
      "  \n",
      "\n",
      "If you found this useful, please cite this write-up as:\n",
      "\n",
      "> Yan, Ziyou. (Jul 2023). Patterns for Building LLM-based Systems & Products.\n",
      "> eugeneyan.com. https://eugeneyan.com/writing/llm-patterns/.\n",
      "\n",
      "or\n",
      "\n",
      "    \n",
      "    \n",
      "    @article{yan2023llm-patterns,\n",
      "      title   = {Patterns for Building LLM-based Systems & Products},\n",
      "      author  = {Yan, Ziyou},\n",
      "      journal = {eugeneyan.com},\n",
      "      year    = {2023},\n",
      "      month   = {Jul},\n",
      "      url     = {https://eugeneyan.com/writing/llm-patterns/}\n",
      "    }\n",
      "\n",
      "  \n",
      "Share on:\n",
      "\n",
      "![](/assets/icon-twitter.svg)\n",
      "\n",
      "![](/assets/icon-linkedin.svg)\n",
      "\n",
      "![](/assets/icon-facebook.svg)\n",
      "\n",
      "![](/assets/icon-mail.svg)\n",
      "\n",
      "  \n",
      "Browse related tags: [ [llm](/tag/llm/) [engineering](/tag/engineering/)\n",
      "[production](/tag/production/) [🔥](/tag/🔥/) ]\n",
      "\n",
      "[ ![](/assets/icon-search.svg)Search](/search/ \"Search\")\n",
      "\n",
      "[« Obsidian-Copilot: An Assistant for Writing & Reflecting](/writing/obsidian-\n",
      "copilot/) [How to Match LLM Patterns to Problems »](/writing/llm-problems/)\n",
      "\n",
      "* * *\n",
      "\n",
      "Join **6,800+** readers getting updates on machine learning, RecSys, LLMs, and\n",
      "engineering.\n",
      "\n",
      "Get email updates\n",
      "\n",
      "* * *\n",
      "\n",
      "  * ![](/assets/icon-twitter.svg) [Twitter](https://twitter.com/eugeneyan \"Twitter\")\n",
      "  * ![](/assets/icon-linkedin.svg) [LinkedIn](https://www.linkedin.com/in/eugeneyan/ \"Linkedin\")\n",
      "  * ![](/assets/icon-threads.svg) [Threads](https://www.threads.net/@eugeneyan \"Threads\")\n",
      "  * ![](/assets/icon-github.svg) [GitHub](https://github.com/eugeneyan/ \"GitHub\")\n",
      "\n",
      "Eugene Yan designs, builds, and operates machine learning systems that serve\n",
      "customers at scale. He's currently a Senior Applied Scientist at Amazon.\n",
      "Previously, he led machine learning at Lazada (acquired by Alibaba) and a\n",
      "Healthtech Series A. He [writes](/writing/) & [speaks](/speaking/) about\n",
      "machine learning, recommenders, LLMs, and engineering at\n",
      "[eugeneyan.com](https://eugeneyan.com/) and\n",
      "[ApplyingML.com](https://applyingml.com/).\n",
      "\n",
      "© Eugene Yan 2015 - 2024 • [Feedback](/site-feedback/) • [RSS](/rss/)\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(docs[0].get_content())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Nodes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "orig_nodes = node_parser.get_nodes_from_documents(docs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "because evals were often conducted with untested, incorrect\n",
      "ROUGE implementations.\n",
      "\n",
      "![Dimensions of model evaluations with ROUGE](/assets/rogue-scores.jpg)\n",
      "\n",
      "Dimensions of model evaluations with ROUGE\n",
      "([source](https://aclanthology.org/2023.acl-long.107/))\n",
      "\n",
      "And even with recent benchmarks such as MMLU, **the same model can get\n",
      "significantly different scores based on the eval implementation**.\n",
      "[Huggingface compared the original MMLU\n",
      "implementation](https://huggingface.co/blog/evaluating-mmlu-leaderboard) with\n",
      "the HELM and EleutherAI implementations and found that the same example could\n",
      "have different prompts across various providers.\n",
      "\n",
      "![Different prompts for the same question across MMLU\n",
      "implementations](/assets/mmlu-prompt.jpg)\n",
      "\n",
      "Different prompts for the same question across MMLU implementations\n",
      "([source](https://huggingface.co/blog/evaluating-mmlu-leaderboard))\n",
      "\n",
      "Furthermore, the evaluation approach differed across all three benchmarks:\n",
      "\n",
      "  * Original MMLU: Compares predicted probabilities on the answers only (A, B, C, D)\n",
      "  * HELM: Uses the next token probabilities from the model and picks the token with the\n"
     ]
    }
   ],
   "source": [
    "print(orig_nodes[20:28][3].get_content(metadata_mode=\"all\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question Extractor on Nodes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 8/8 [00:03<00:00,  2.04it/s]\n"
     ]
    }
   ],
   "source": [
    "nodes_1 = node_parser.get_nodes_from_documents(docs)[20:28]\n",
    "nodes_1 = question_extractor(nodes_1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Excerpt from document]\n",
      "questions_this_excerpt_can_answer: 1. How do different implementations of the MMLU benchmark affect the scores of the same model?\n",
      "2. What are the differences in evaluation approaches between the original MMLU benchmark, HELM, and EleutherAI implementations?\n",
      "3. How do various providers differ in the prompts they use for evaluating models in the MMLU benchmark?\n",
      "Excerpt:\n",
      "-----\n",
      "because evals were often conducted with untested, incorrect\n",
      "ROUGE implementations.\n",
      "\n",
      "![Dimensions of model evaluations with ROUGE](/assets/rogue-scores.jpg)\n",
      "\n",
      "Dimensions of model evaluations with ROUGE\n",
      "([source](https://aclanthology.org/2023.acl-long.107/))\n",
      "\n",
      "And even with recent benchmarks such as MMLU, **the same model can get\n",
      "significantly different scores based on the eval implementation**.\n",
      "[Huggingface compared the original MMLU\n",
      "implementation](https://huggingface.co/blog/evaluating-mmlu-leaderboard) with\n",
      "the HELM and EleutherAI implementations and found that the same example could\n",
      "have different prompts across various providers.\n",
      "\n",
      "![Different prompts for the same question across MMLU\n",
      "implementations](/assets/mmlu-prompt.jpg)\n",
      "\n",
      "Different prompts for the same question across MMLU implementations\n",
      "([source](https://huggingface.co/blog/evaluating-mmlu-leaderboard))\n",
      "\n",
      "Furthermore, the evaluation approach differed across all three benchmarks:\n",
      "\n",
      "  * Original MMLU: Compares predicted probabilities on the answers only (A, B, C, D)\n",
      "  * HELM: Uses the next token probabilities from the model and picks the token with the\n",
      "-----\n"
     ]
    }
   ],
   "source": [
    "print(nodes_1[3].get_content(metadata_mode=\"all\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build Indices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.core import VectorStoreIndex\n",
    "from llama_index.core.response.notebook_utils import (\n",
    "    display_source_node,\n",
    "    display_response,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "index0 = VectorStoreIndex(orig_nodes)\n",
    "index1 = VectorStoreIndex(orig_nodes[:20] + nodes_1 + orig_nodes[28:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Query Engines"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "query_engine0 = index0.as_query_engine(similarity_top_k=1)\n",
    "query_engine1 = index1.as_query_engine(similarity_top_k=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Querying"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "query_str = (\n",
    "    \"Can you describe metrics for evaluating text generation quality, compare\"\n",
    "    \" them, and tell me about their downsides\"\n",
    ")\n",
    "\n",
    "response0 = query_engine0.query(query_str)\n",
    "response1 = query_engine1.query(query_str)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "**`Final Response:`** Metrics for evaluating text generation quality can be categorized as context-dependent or context-free. Context-dependent metrics consider the context of the task and may need adjustments for different tasks. On the other hand, context-free metrics do not consider task-specific context and are easier to apply across various tasks.\n",
       "\n",
       "Some commonly used metrics for evaluating text generation quality include BLEU, ROUGE, BERTScore, and MoverScore. \n",
       "\n",
       "- **BLEU (Bilingual Evaluation Understudy)** is a precision-based metric that compares n-grams in the generated output with those in the reference. \n",
       "- **ROUGE (Recall-Oriented Understudy for Gisting Evaluation)** evaluates the overlap between the generated output and reference summaries.\n",
       "- **BERTScore** leverages contextual embeddings to measure the similarity between the generated output and reference.\n",
       "- **MoverScore** considers the semantic similarity between the generated output and reference using Earth Mover's Distance.\n",
       "\n",
       "Each of these metrics has its own strengths and weaknesses. For example, BLEU may not capture the overall fluency and coherence of the generated text, while ROUGE may not consider the semantic meaning adequately. BERTScore and MoverScore, on the other hand, may require pre-trained models and can be computationally expensive. It's important to consider the specific requirements of the task when selecting an appropriate evaluation metric."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "---"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "**`Source Node 1/1`**"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "**Node ID:** 4edc4466-e9ae-47ae-b0ee-8a8ac27a0378<br>**Similarity:** 0.8381672789063448<br>**Text:** GPT-4) prefers the output of one model over a reference model. Metrics include win rate, bias, latency, price, variance, etc. Validated to have high agreement with 20k human annotations.\n",
       "\n",
       "We can group metrics into two categories: context-dependent or context-free.\n",
       "\n",
       "  * **Context-dependent** : These take context into account. They’re often proposed for a specific task; repurposing them for other tasks will require some adjustment.\n",
       "  * **Context-free** : These aren’t tied to the context when evaluating generated output; they only compare the output with the provided gold references. As they’re task agnostic, they’re easier to apply to a wide variety of tasks.\n",
       "\n",
       "To get a better sense of these metrics (and their potential shortfalls), we’ll\n",
       "explore a few of the commonly used metrics such as BLEU, ROUGE, BERTScore, and\n",
       "MoverScore.\n",
       "\n",
       "**[BLEU](https://dl.acm.org/doi/10.3115/1073083.1073135) (Bilingual Evaluation\n",
       "Understudy)** is a precision-based metric: It counts the number of n-grams in\n",
       "th...<br>**Metadata:** {}<br>"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display_response(\n",
    "    response0, source_length=1000, show_source=True, show_source_metadata=True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "**`Final Response:`** Metrics for evaluating text generation quality include BLEU and ROUGE. These metrics are commonly used but have limitations. BLEU and ROUGE have shown poor correlation with human judgments in terms of fluency and adequacy. They also exhibit low correlation with tasks that require creativity and diversity in text generation. Additionally, exact match metrics like BLEU and ROUGE are not suitable for tasks such as abstractive summarization or dialogue in text generation due to their reliance on n-gram overlap, which may not capture the nuances of these tasks effectively."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "---"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "**`Source Node 1/1`**"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "**Node ID:** 52856a1d-be29-494a-84be-e8db8a736675<br>**Similarity:** 0.8459422950143721<br>**Text:** finds the minimum effort to transform\n",
       "one text into another. The idea is to measure the distance that words would\n",
       "have to move to convert one sequence to another.\n",
       "\n",
       "However, there are several pitfalls to using these conventional benchmarks and\n",
       "metrics.\n",
       "\n",
       "First, there’s **poor correlation between these metrics and human judgments.**\n",
       "BLEU, ROUGE, and others have had [negative correlation with how humans\n",
       "evaluate fluency](https://arxiv.org/abs/2008.12009). They also showed moderate\n",
       "to less correlation with human adequacy scores. In particular, BLEU and ROUGE\n",
       "have [low correlation with tasks that require creativity and\n",
       "diversity](https://arxiv.org/abs/2303.16634).\n",
       "\n",
       "Second, these metrics often have **poor adaptability to a wider variety of\n",
       "tasks**. Adopting a metric proposed for one task to another is not always\n",
       "prudent. For example, exact match metrics such as BLEU and ROUGE are a poor\n",
       "fit for tasks like abstractive summarization or dialogue. Since they’re based\n",
       "on n-gram overlap between ...<br>**Metadata:** {'questions_this_excerpt_can_answer': '1. How do conventional benchmarks and metrics for measuring text transformation performance compare to human judgments in terms of fluency and adequacy?\\n2. What is the correlation between metrics like BLEU and ROUGE and tasks that require creativity and diversity in text generation?\\n3. Why are exact match metrics like BLEU and ROUGE not suitable for tasks like abstractive summarization or dialogue in text generation?'}<br>"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display_response(\n",
    "    response1, source_length=1000, show_source=True, show_source_metadata=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extract Metadata Using PydanticProgramExtractor\n",
    "\n",
    "PydanticProgramExtractor enables extracting an entire Pydantic object using an LLM.\n",
    "\n",
    "This approach allows for extracting multiple entities in a single LLM call, offering an advantage over using a single metadata extractor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pydantic import BaseModel, Field\n",
    "from typing import List"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setup the Pydantic Model¶\n",
    "\n",
    "Here we define a basic structured schema that we want to extract. It contains:\n",
    "\n",
    "Entities: unique entities in a text chunk\n",
    "Summary: a concise summary of the text chunk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class NodeMetadata(BaseModel):\n",
    "    \"\"\"Node metadata.\"\"\"\n",
    "\n",
    "    entities: List[str] = Field(\n",
    "        ..., description=\"Unique entities in this text chunk.\"\n",
    "    )\n",
    "    summary: str = Field(\n",
    "        ..., description=\"A concise summary of this text chunk.\"\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setup the Extractor¶\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.program.openai import OpenAIPydanticProgram\n",
    "from llama_index.core.extractors import PydanticProgramExtractor\n",
    "\n",
    "EXTRACT_TEMPLATE_STR = \"\"\"\\\n",
    "Here is the content of the section:\n",
    "----------------\n",
    "{context_str}\n",
    "----------------\n",
    "Given the contextual information, extract out a {class_name} object.\\\n",
    "\"\"\"\n",
    "\n",
    "openai_program = OpenAIPydanticProgram.from_defaults(\n",
    "    output_cls=NodeMetadata,\n",
    "    prompt_template_str=\"{input}\",\n",
    "    extract_template_str=EXTRACT_TEMPLATE_STR,\n",
    ")\n",
    "\n",
    "metadata_extractor = PydanticProgramExtractor(\n",
    "    program=openai_program, input_key=\"input\", show_progress=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Extract metadata from the node"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 1/1 [00:01<00:00,  1.51s/it]\n"
     ]
    }
   ],
   "source": [
    "extract_metadata = metadata_extractor.extract(orig_nodes[0:1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'entities': ['eugeneyan', 'llm', 'engineering', 'production'],\n",
       "  'summary': 'Patterns for Building LLM-based Systems & Products - Discussions on HackerNews, Twitter, and LinkedIn. There is a large class of problems that are easy to imagine and build demos for, but extremely hard to make products out of. For example, self-driving: It’s easy to demo a'}]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "extract_metadata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 1/1 [00:01<00:00,  1.03s/it]\n"
     ]
    }
   ],
   "source": [
    "metadata_nodes = metadata_extractor.process_nodes(orig_nodes[0:1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[TextNode(id_='2b6a40a8-dd6a-44a8-a005-da32ad98a05c', embedding=None, metadata={'entities': ['eugeneyan', 'llm', 'engineering', 'production'], 'summary': 'Patterns for Building LLM-based Systems & Products - Discussions on HackerNews, Twitter, and LinkedIn. Content includes discussions on self-driving technology and challenges in turning demos into products.'}, excluded_embed_metadata_keys=[], excluded_llm_metadata_keys=[], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='https://eugeneyan.com/writing/llm-patterns/', node_type=<ObjectType.DOCUMENT: '4'>, metadata={}, hash='9da2827b0860b2f81e51cb3efd93a13227f0e4312355a495e5668669f257cb14'), <NodeRelationship.NEXT: '3'>: RelatedNodeInfo(node_id='d3a86dba-7579-4196-80d7-30affa7052a7', node_type=<ObjectType.TEXT: '1'>, metadata={}, hash='993e43bb060cf2f183f894f8dec6708eadcac2b7d2760a94916dc82c24255acc')}, text='# [eugeneyan](/)\\n\\n  * [Start Here](/start-here/ \"Start Here\")\\n  * [Writing](/writing/ \"Writing\")\\n  * [Speaking](/speaking/ \"Speaking\")\\n  * [Prototyping](/prototyping/ \"Prototyping\")\\n  * [About](/about/ \"About\")\\n\\n# Patterns for Building LLM-based Systems & Products\\n\\n[ [llm](/tag/llm/) [engineering](/tag/engineering/)\\n[production](/tag/production/) [🔥](/tag/🔥/) ]  · 66 min read\\n\\n> Discussions on [HackerNews](https://news.ycombinator.com/item?id=36965993),\\n> [Twitter](https://twitter.com/eugeneyan/status/1686531758701899776), and\\n> [LinkedIn](https://www.linkedin.com/posts/eugeneyan_patterns-for-building-\\n> llm-based-systems-activity-7092300473981927424-_wVo)\\n\\n“There is a large class of problems that are easy to imagine and build demos\\nfor, but extremely hard to make products out of. For example, self-driving:\\nIt’s easy to demo a', start_char_idx=0, end_char_idx=838, text_template='[Excerpt from document]\\n{metadata_str}\\nExcerpt:\\n-----\\n{content}\\n-----\\n', metadata_template='{key}: {value}', metadata_seperator='\\n')]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metadata_nodes"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  },
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
